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ABSTRACT 

A project studied use of state unemployment insurance 
(UI) wage-record data for Program Year 1986 to assess the 
effectiveness of Job Training Partnership Act (JTPA) programs. Eleven 
of 27 interested states were selected by ability to meet data 
requirements. These issues associated with the use of UI data as an 
assessment tool were expl-^red: coverage, accuracy, timeliness, and 
confidentiality. Among the principal advantages of wage record data 
for program analyses identified were flexibility, capacity to provide 
longitudinal data, usefulness as a resource for net impact studies, 
and cost-effectiveness. A database containing longitudinal UI 
waje-record data and JTPA program data was developed. A comparison of 
wage-record data with post-program survey data found that on each of 
three criteria for judging data quality (lack of bias, precision, and 
consistency), wage record riata were superior* An examination of pre^ 
and post-program employment and earnings of JTPA participants 
indicated increased post-program earnings but declining post-program 
employment and a delay in increased employment for participants in 
remedial and basic education. Roles for wage record data were 
suggested in the context of the JTPA performance standard system: 
identification of groups of participants with limited employment 
experience and development of post-program performance measures, 
(Twenty-three references and 32 tables/figures are included* 
Appendixes contain data tables and lists of data file elements.) 
(YLB) 



i 




(Je 7"his clocumenf has hA*n 

O Minor Channel hiv* . 

'eproducl.on qu.r;* "^"^ ™P'o«e 



.'^^f^|:;.^«r^^p^ ^ ^^^^^ ^ 




If 

1 



A Feasibility Study of the 
Use of Unemployment 
Insurance Wage-Record 
Data as an Evaluation 
Tool for JTPA 

Report on Project's Phase I Activities 

Research Report Number 90-02 

John Baj and Charles E. Trott 
The Center for Governmental Studies 
Northern Illinois University 

with 

David Stevens 
University of Maryland Baltimore County 

National Commission for Employment Policy 

January 1991 



•yhis project was conducted with funds provided under 
programs of the Job Training Partnership Act. The funds 
were provided by the National Commission for Employment 
Policy to the Illinois Department of Commerce and 
Community Affairs. The positions reported in this paper do 
not necessarily represent in whole or in part the viewpoint of 
the National Commission for Employment Policy or the 
Illinois Department of Commerce and Community Affairs. 



•1 



Acknov/ledgements 



A study as large and potentially significant as this one is always the result of the 

efforts of many people. A list of the participants from each of the 11 states is 
provided on a separate page. We extend our thanks to all of them for their efforts and 
encouragement. In turn, we urge them to continue to use their capacity to match JTPA 
and UI data in managing and evaluating their JTPA programs. 

Singling out individuals for special recognition is risky for fear of overlooking one or 
more who were especially helpful in the project. Three people, however, have given 
tirelessly of their assistance to us in the first phase of this project. Ray Worden has been 
a most insightful, positively demanding, and encouraging project manager. With 
determination and good humor, he has provided invaluable guidance in the design and 
direction of this feasibility study. 

We would also like to thank David Stevens and Tim Harmon for being on call when 
we needed them and for so willingly critiquing our thoughts, analyses, and v^iting. 
Their assistance has been timely and insightful. 

Many people reviewed drafts of this report and served to provide us with instructive 
comments, questions, and guidapce. Especially helpful were the comments and queries 
from Karen Greene and her sta« in the Office of Strategic Planning and Policy 
Development, Employment and Training Administration, U.S. Department of Labor. 
In addition, we want to thank Vince Geraci of the National Commission for 
Employment Policy for his critical and positive insights. 

Ultimate responsibility for the content of this report is ours. We hope our mistakes 
are few and minor, and that the importance of this endeavor and our findings will have 
the significant impact we think they deserve. 



T jhn Baj and Charles E. Trott 
Center ror Governmental Studies 
Northern Illinois University 
DeKalb, Illinois 



Preface 



Jn May of 1988 the National Commission for Employment Policy launched a 

multi-state project to study the feasibility of using unemployment insurance (UI) 
wage records for the evaluation of programs funded under the Job Training 
Partnership Act (JTPA). Such evaluations would address the legislative provision that 
job training is an investment in human capital and that the return on this investment 
shall be measured in terms of increases in the participants' employment and earnings 
and reductions in welfare dependency. This report covers the first phase of the project. 
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In March 1988, the National Commission 
for Employment Pobcy invited all states to 
participate in a project designed to explore 
the issues involved 'n using state 
unemployment insurance (UI) wage-record 
data to assess the effectiveness of JTPA 
programs. The Commission was interested 
in exanuning these issues across states to 
evaluate the feasibility of using Ul data as a 
national evaluation tool. Twenty-seven 
states expressed interest in this request; 
eleven states were selected on the basis of 
their ability to meet the project's data 
requirements: Florida, Idaho, Illinois, 
Indiana, Missouri, Nevada, Oregon, South 
Carolina, Utah, Virginia, and Washington. 
The state of Illinois was also selected as the 
clearinghouse for the project. Under its 
auspices, the Center for Governmental 
Studies at Northern Illinois University 
served to compile and analyze the 
multi-state database. 

In conjunction with this feasibility 
assessment, the Comnussion outlined four 
major analytical goals for the project: 

• To study both pre-JTPA and 
post-JTPA earnings patterns of 
individuils who terminated from 
JTPA during Program Year 1986 
(PY86"July 1, 1986 to June 30, 1987). 

• To explore the differences in 
earnings by subgroups of 
participants. 

• To compare earnings patterns by 
type of service(s) provided. 



• To learn how the results of the 
13-week participant follow-up 
begun in PY86 compare to the 
post-UI data. 

This report presents the results of the Prst 
phase of the feasibility assessment and 
summarizes the findings of the research 
performed to date to achieve the arwlytical 
goals. 

Subsequent phases of this project will 
increase the number of states and add data 
on terminees from Program Years 1987 and 
1988. Analyses of these expanded data sets 
should serve to clarify and extend the 
findings summarized below. 

Major Findings 

THE UNEMPLOYMENT INSURANCE 
WAGE-RECORD DATA 
COLLECTION SYSTEM 

The Deficit Reduction Act of 1984 
increased the level of uniformity among 
state systems for collecting UI data. Before 
the legislation, some states had adopted a 
wage request approach to operate their UI 
systems. This approach required employers 
to submit wage information only when it 
was necessary to process an unemployment 
insurance claim. The Deficit Reduction Act 
eliminated the wage request option. As of 
September 30, 1988, the vast majority of 
employers in all states were required to 
submit quarterly information on earnings. 

Technical discussions of the potential use 
of Ul wage records for longitudinal analysis 
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typically center on fo ir topics: coverage, 
accuracy, timeliness, and confidentiality. 
An analysis of these issues indicated the 
fo jv^ng: 

• Virtually all jobs that moit observers 
would consider appropriate targets 
for JTPA terminee placement are 
covered by the UI reporting systei 

• Nearly all of these jobs offer money 
v^ages, commissions, and bonus 
forms of Compensation, which are 
subject to accurate reporting 
practices. 

• The incidence of late reporting by 
employers is low and expected to 
improve as more businesses adopt 
electronic reporting procedures. 

• State laws and administrative 
regulations need not act as a barrier 
to the responsible use of the 
wage-record data in full compliance 
with the spirit and case law of state 
privacy requirements. 

THE UNEMPLOYMENT INSURANCE 
WAGE-RECORD SYS l EM AS AN 
ASSESSMENT TOOL 

Because each UI quarterly wage record 
contains information from each employer 
about each employee, the information can be 
analyzed in a variety of ways. When JTPA 
records are merged with UI wage records, 
post-program earnings histories can be 
examined by participant characteristics 
(age, sex, etc.), types of program 
interventions (basic educational training, 
on-the-job training, etc.), service provider, 
geography (county of residence, SDA, state), 
industry of pre-program employment, 
industry of post-program employment, or 
any combination of the above. 



UI wage records also permit the tracking of 
post-program experiences well beyond the 
current 13- week post-program period. By 
.ontinually appending additional quarters 
of wage information to client records, one 
can follow the progress of JTPA terminees 
for several years if desired. In addition, the 
use of the UI wage records could 
substantially expand the pre-program 
information base on JTPA clients without 
imposing the burden of collecting additional 
data on local program operators. 

The UI system is an excellent resource for 
studies concerned with assessing the net 
impact of JTPA and other employment and 
training programs. Since UI data can be 
obtained both for program participants and 
members of a comparison or control group, 
the employment and earnings outjcmes of 
the two groups can be defined and 
measured consistently. Finally, UI 
wage-record dala are an extremely 
cost-effective alternative to othei sources of 
post-program information. 

This review of features of the UI 
wage-record system as an assessment tool 
revealed the foUowin'*. 

• The use of UI wage-record da^a 
would enable detailed analyses of 
the long-term employment and 
earnings patterns of JTPA terminees. 

• UI wage-record data offer great 
promise as a resource for studies 
designed to measure the net impact 
of employment and training 
programs. 

• UI wage-record data are a 
cost-effective means for obtaining 
pre- and post-program employment 
and earnings information on JTPA 
participants. 
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THE DEVELOPMENT OF THE 
11 -STATE DATABASE 

One of the major activities of the first phase 
of this project involved the construction of a 
database containing longitudinal UI 
wage-record data and JTPA program data 
from the 1 1 participating states. The effort 
required each state to negotiate agreements 
for the exchange of data within the state and 
with the Commission. Each state also had to 
match the JTPA and Ul files and produce a 
set of standard tables. The clearinghouse 
was required to establish data transfer 
conventions for all states to follow^ compile 
Stat? files, check them against the standard 
tables the states produced, and merge the 
state files to produce a unified database. 
While there were some technical issues, 
most of the problenis were encountered in 
processing the JTPA files rather than the UI 
files. However, these problen\s were minor 
and, once identified, were quickly corrected. 

The project paid particular attention to the 
activity or intervention records maintained 
in the state JTPA management iiiformation 
systems. Activity data were particularly 
prone to coding errors, and some service 
delivery areas (SDAs) may under-report this 
information. 

The experience of constructing the 1 1 -state 
database of matched Ul wage records and 
JTPA terminee and program intervention 
records is summarized as follows: 

• The technical problems of matching 
records from the two data systems 
are relatively minor and easily 

overcome. 

• If a state resolves to use the UI 
wage-record information as an 
assessment tool, necessary 
agreements and procedures for 



exchanging data can be established 
and executed. 

• There are a number of specific issues 
regarding the accuracy and 
reliability of program activity data. 

COMPARISON BETWEEN Ul 
WAGE-RECORD DATA AND THE 
POST-PROGRAM DATA OBTAINED 
FROM SURVEYING JTPA 
PARTICIPANTS 

The database constructed in the first phase 
of this project provided an opportunity to 
examine some of the issues surroimding the 
quality of the post-program survey data. 
The study revealed that certain subgroups 
of the JTPA population had lower response 
rates even aJfter the effects of employment 
status at termination were controlled. In 
particular, males, minorities, high school 
dropouts, and welfare recipients had 
significantly lower response rates to the 
survey. This evidence suggests that the 
levels of service ar SDA provides to these 
groups probably influences the respoi\se 
rates it achieves on the post-program 
survey. 

The project's database also provided the 
first large-scale opportunity to examine the 
issue of nonrespor\se bias and its impact on 
the i-stimates of program performance. The 
study found that the post-program 
employment experiences of survey 
respondents and nonrespondents were 
distinctly different. In particular, 
respondents had higher post-program 
employment rates than nonrespondents, 
suggesting that the survey estimates of 
post-program employment are upwardly 
biased. Further analysis revealed the source 
of this bias and that the current nonresponse 
adjustment procedures were ineffective in 
combatting this problem. 
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Overall, there was a fairly high degree of 
correspondence between the employment 
information obtained from respondents to 
the survey and their UI records. Efforts 
were made to identify the roots of the 
discrepancies between the two data sources. 
Although data limitations inhibited this 
analysis, the evidence suggests that two 
explanations are likely to account for the 
majority of these mismatches, namely, 
out-of-state employment and noncovered 
employment, e.g., self-employment. In both 
cases, the problems associated with these 
coverage issues can be addressed through 
several remedial measures. 

The relative costs of the two data collection 
options were examined. The study roughly 
estimated that if each state in the nation 
implemented a Ul-based data collection 
system, the total start-up cost would be less 
than one-half of the current cost of collecting 
post-piogram survey data. Furthermore, 
the ongoing costs would be less than 
one-fifth of the cost of participant contact. 
This could result in a saving of over two 
million dollars each program year. Giver 
the greater volume of information that can 
be accessed from state UI systems, these 
preliminary estimates understate the cost 
effectiveness of the Ul-based approach. 

The major findings of this comparison 
between the two data sources areas follows: 

- There is evidence of nonresponse 
bias in the 13- week post-program 
survey data that inflates the 
estimates of post-program 
performance. 

• Although there are coverage issues 
surrounding the use of UI 
wage-record data, they are unhkely 
to pose insurmountable barriers to 
the confident use of this data for 
performance assessment. 



• The cost of using UI wage-record 
data can be expected to be much 
lower than current survey costs and 
may actually decline. 

SUBGROUP AND ACTIVITY 
ANALYSES USING UI 
WAGE-RECORD DATA 

The project performed several analyses to 
demonstrate the different ways UI 
wage-record data can be used to examine the 
employment and eanungs experiences of 
JTPA participants. The first set of analyses 
explored the pre- and post-program 
employment and earnings trends of various 
demographic subgroups in the JTPA 
population. The second set of analyses 
considered the experiences of participants 
grouped on the basis of the type of 
interventions they received during their 
tenure in the JTPA program. The study 
examined both the distribution of 
participants across various sequences of 
program activities and the termination 
outcomes for those sequences. It also 
examined the pre- and post-program 
employment and earnings of participants 
who received the most common JTPA 
services. 

Although the analyses should be viewed as 
exploratory, the findings provide some 
insights into the JTPA program. The major 
findings to date include the following: 

• Although the earnings of JTPA 
participants show a sharp decline 
prior to enrollment, employment 
levels remain fairly constant, 

• The average post-program earnings 
of terminees increase during the 
post-program period, but 
employment levels decline. 
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• Terminees with lower pre-program 
earnings and employment 
experiences tend to receive more 
intensive JT? A services. 

• The differentials found in the 
post-program earnings of 
participants receiving different 
types of intervention tend to 
correspond to the differences found 
in their pre-program earnings. 

• There is some evidence that remedial 
and basic education lead to increases 
in employment for participants 
which show up after a delay of a few 
quarters upon termination from this 
activity. 

THE ROLE OF Ul WAGE-RECORD 
DATA IN THE CONTEXT OF THE 
PERFORMANCE STANDARDS 
SYSTEM 

Wagerecord data can play two important 
roles in the context of the JTPA performance 
standards system. The first role is to define 
groups of participants with limited 
employment experience. Once these groups 
are identified, efforts can be made to provide 
performance standards adjustments to 
SD As based on their levels of service to these 
groups. The evidence indicates that such 
groups can be defined and that performance 
standards adjustments are warranted- 

Wage-record data can also be used to 
develop a variety of post-program 
performance measures. This report 
examines and discusses a number of 
candidate post-program measures. Of 
special note is that UI wage-record data can 
be used to capture the level of post-program 
employment retention among JTPA 
participants. 



The major findings to date concerning the 
possible role of UI wage-record data in the 
performance standards system can be 
summarized as follows: 

• The preliminary evidence indicates 
that pre-program UI information can 
effectively be used to isolate 
hard-to-serve client groups and 
provide performance standards 
adjustments to SDAs for serving 
these populations, 

• The time lag involved with gaining 
access to Ul wage-record data is not 
viewed as a nwjor barrier to the use 
of this information in the 
performance standards system. 

• Ul information can be used to design 
a variety of post-program 
performance measures which more 
adequately reflect the goals of the 
Job Training Partnership Act than 
the current survey-based measures, 
e.g./ long-term earnings and 
employment retention. 

IMPLICATIONS FOR THE 
JTPA SYSTEM 

The findings from the first phase of this 
project indicate that JTPA and any other 
program whose goal is to increase the 
employment and earnings of participants 
can use Ul wage-record data with 
confidence. Obtaining post-program 
information from state UI systenns is not 
only a viable option, it is far more 
cost-effective than the current practice of 
gathering this information through contact 
with participants. Furthermore, UI dala are 
of higher quality than corresponding 
survey-based information. In short, UI 
wage-record data can be used to develop an 
extremely flexible, longitudinal database 
representing virtually all JTPA participants, 
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which will greatly enhance the ability of the 
Congress to evaluate the performance of the 
JTPA program. 

These findings suggest that further steps 
should be taken to facilitate the movement 
of JTPA towards a post-program assessment 
system based on UI data. This change 
would affect all levels of the 
program-federal, state, and local. As a 
result, it is necessary to examine the 
implications at each of these levels. 

FEDERAL LEVEL 

One of the major implications of moving to 
a post-program assessment system based on 
UI data is the need to create new 
performance measures. The fact that UI 
information can be used to track individuals 
for multiple quarters after program 
participation offers a great deal of flexibility 
in designing these measures. This report 
illustrates the potential for using UI data to 
develop long-term post-program measures 
that can tap different dimensions of the 
employment and earnings experiences of 
JTPA participants. 

A second implication of moving to an 
assessment system based on UI data is the 
need to develop a transitional strategy to 
allow states enough time to adjust to the new 
measures. One option is to adopt the 
strategy used to introduce the post-program 
performance measures based on survey 
data. That strategy allowed each state to 
select from a menu of alternative 
performance standards. This menu 
approach proved to be very effective then, 
and there is no reason to believe that it 
would be less effective now. 

A final implication concerns the pn blem of 
the time lag involved in gaining acc ■ -s to UI 
information. DOL confronted this issue 
before in the movement from 



termination-based performance measures 
to the current post-program measures. The 
solution was to begin the performance 
measurement period in the last quarter of 
the program year under evaluation. The 
same approach can be used to accommodate 
the slightly greater time lag imposed by 
using UI wage-record information. 

STATE LEVEL 

The first and most obvious implication of 
moving to a Ul-based assessment system for 
a state is that the agency administering JTPA 
will need to obtain UI wage-record data. 
Many state JTPA agencies, including those 
participating in this project, have already 
developed procedures to gain acces" to this 
information. However, a number of other 
states will need to forge data exchange 
agreements to obtain the UI data. A state's 
ability to accomplish this task is likely to 
depend on the current level of integration 
and coordination between its JTPA and UI 
systems. In those states where the level of 
contact has been minimal, the process of 
developing these agreements may foster 
closer working ties between the two 
systems. 

A state will also need to decide how to 
address the issue of out-of-state 
employment. There are a number of 
alternatives. A state may decide to rely on a 
statistical adjustment procedure. 
Alternatively, it may conduct a 
supplemental post-program survey for 
those terminees who were placed in 
out-of-state employment. A state may also 
decide to develop data-sharing agreements 
with neighboring states. Interest in 
developing a national archive for UI data is 
increasing. If such an archive is developed, 
the coverage issue surrounding out-of-state 
employment will become moot. 
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Executive Summary 



Once access to UI data is obtained, states 
will confront the technical problem of 
merging UI information with their cvurent 
JTPA management information systems 
(MIS). States will also need to decide how 
long they wish to track the progress of JTPA 
terminees given the limits of their \4IS to 
store information. 

SDA LEVEL 

For many SDAs, EXDL's change in PY90 to 
a system that bases adult and welfare 
performance standards solely on 
post-program survey data has created a 
difficult management environment. In the 
past, SDAs have used the performance 
measures to monitor the performance of 
their service providers. However, this 
management tool loses its effectiveness 
when performance measures are limited to 
post-program survey data. In the majority 
of states, post-program data are collected 
from only a sample of the SDA's former 
participants. Consequently, only a very 
small number of terminees served by a given 
subcontractor are likely to be selected for 
post-program follow-up. The number of 
clients sampled from each subcontractor is 
unlikely to be large enough to permit 
confident evaluations of subcontractor 
performance. This problem is especially 
acute in large urban SDAs that have an 
extensive network of providers to monitor. 

The movement to an assessment system 
based on UI data would return this 
management tool to SDAs. Through the UI 
system, post-program information can be 
obtained for virtually all of an SDA's 
terminees. An SDA would be able to use this 
information to assess the performance of 
subcontractors on the same measures the 
state uses to assess ;ne SDA's performance. 
The results of these assessments could then 
be used as a basis for future contract 
negotiations. 



FINAL IMPLICATIONS 

The use of UI wage-record daia as an 
evaluation tool has implications beyond the 
JTPA program. Increasingly, Congress has 
turned to the use of performance measures 
to insure greater accountability in federally 
funded progranw and to monitor their 
performance. Recent national welfare and 
vocational education legislation are just two 
examples of this trend. In addition, the 
effort to improve the current levels of 
coordination and cooperation among these 
programs is likely to produce higher levels 
of co-eiu-ollment in the future. These two 
factors, performance measures and 
co-enrollment, argue for a common 
framework to measure labor force outcomes 
of employment and training programs. 
Methods of collecting data based on surveys 
are ill-suited to establishing such a 
framework. They require extraordinary 
efforts to impose some consistency among 
the various survey efforts and to avoid 
duplication of effort. Furthermore, each 
program may wish to examine the progress 
of former clients using different 
post-program time frames. This would 
diminish the effectiveness of any 
coordination effort, and many clients who 
are co-enrolled will likely face the prospect 
of being the subject of multiple surveys. 

On the other hand, a coordinated UI data 
collection effort providing consistent and 
reliable data for all programs is a distinct 
possibility. Such an effort will allow each 
program to choose its own post-program 
measurement period and still benefit from a 
cost-sharing approach. As federal funds 
become increasingly scarce, greater efforts 
should be made to promote the use of the 
inexpensive and high-quality information 
waiting to be accessed on state UI data 
systems. 
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Chapter 1 
Introduction 



Background 



This study is the product of a long and 
abiding interest within the employment and 
training community concerning the 
possibility of using wage-record data from 
the Unemployment ^-Tsurance System (Ul) 
as a rational evaluation tool. In the past, the 
prospects of using this data source for 
national studies we-'e limited because the 
states had different UI reporting 
requirements. However, recent legislation 
mandating all states to collect quarterly 
wage-record information as of October 1988 
has essentially eliminated this barrier and 
has renewed iiiterest in using UI data for 
program evaluation. The possibility of 
conducting cost-effective program 
evaluations with information gathered by 
state UI systems has also gained the 
attention of Congress. In recent sessions, 
members ot Congress have shown an 
increasing willingness to require or allow 
the use of this information for a variety of 
purposes. For example, pending legislation 
to amend the Job Training Partnership Act 
and to reauthorize the Francis Perkins Act 
contains provisions that permit the use of UI 
records to track participants and evaluate 
program effects. 

In response to these recent developments, 
the National Commission for Employment 
Policy announced a project in late March of 
1988 to study the utihty of UI data for 
assessing the effectiveness of JTPA 
programs. The primary goal of this project 



was to examine the issues involved in using 
this data source for the purpose of 
long-term, post-program evaluation. 

The Commission proposed four major uses 
for UI wage record files: 

• To study both pre-JTPA and 
post-JTPA earnings patterns of 
individuals who terminated from 
JTPA during Program Year 1986.^ 

• To explore the differences in 
earnings by subgroups of 
participants. 

• To compare earnings patterns by 
type of service(s) provided. 

• To compare the results of the 
follow-up study obtained with UI 
data to results of the 13-week 
participant follow-up begun in 1986. 

The Commission encouraged all states to 
submit requests to participate in the study, 
and 27 states responded. Applicants were 
primarily screened according to their ability 
to meet the project's data requirements. 
States had to have JTPA records on 
individual terminees from PY86, including 
records of services received, and UI 
wage-record data for each of these terminees 
dating from at least July 1985. Eleven states 
were selected: Florida, Idaho, Illinois, 
Indiana, Missouri, Nevada, Oregon, Sovth 
Carolina, Utah, Virginia, and Washington. 
The state of Illinois was also selected as the 
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clearinghouse for the project and through 
the Center for Governmental Studies at 
Northern Illinois University served to 
compile and analyze the multistate 
database.^ 

The Commission convened a panel of 
technical experts drawn primarily from 
leading private-sector research and 
consulting firms with extensive experience 
in the quantitative analysis of employment 
and training programs. The panel was 
asked to review the objectives of the study 
and guide its design and implementation. 
The panel voiced two general concerns. 
Hrst/ there was the concern that the pre- and 
post-earnings changes examined in this 
study would be misinterpreted as evidence 
of the net impact of the JTPA program. 
None of the analyses presented in this report 
constitute a study of net impacts, which is a 
more complicated form of analysis designed 
to determine the contribution of program 
participation to the observed gross effects. 
A secondary concern of the panel focused on 
whether the various state MIS systems for 
JTPA management were sufficiently 
consistent and reliable to study the effects of 
sequential or concurrent program activities. 

The panel concluded that the merits of the 
proposed study significantly outweighed its 
concerns, noting in particular three features 
of a JTPA-UI matched file system: 

• Such a matched file may provide the 
only realistic vehicle to address the 
mandate of Section 106 of the Job 
Training Partnership Act, namely to 
measure the increased earnings of 
participants. Even more to the poiiit, 
the matched file approach may be 
the only cost-effective way to satisfy 
the mandate. 

• The matched file approach would 
provide extremely useful 



comparisons with the required 
1 3-week folio w-up telephonesurvey 
data. 

• The successhil ability to match JTPA 
MIS records with UI wage records 
should encourage states to develop a 
long-term evaluation system. 

This report, prepared by the clearinghouse, 
describes the activities, problems, findings, 
and prospects gleaned from the first phase 
of this study to assess the feasibility of using 
wage-record files from the Unemployment 
Insurance system to assess the effects of the 
programs of the Job Training Partnership 
Act. 

Organization of the Report 

This study involved three major activities. 
The first concerned constructing the 
nwtched files in each of the 11 slates and 
merging them into one file. The report will 
discuss the problems associated with 
developing these files. The second activity 
centered on comparing post-program 
records based on UI data to the information 
from the follow-up telephone interviews of 
a sample of terminees. We will discuss the 
results of the comparisons for the eleven 
states and compare the costs of these two 
information sources. The third activity 
concerned actually using the matched files 
to assess program features or results. We 
will describe applications that illustrate the 
utility of the matched file approach for 
cost-effective program evaluation. 

Chapter 2 of the report gives the 
background of the UI system and discusses 
the general issues associated with the use of 
UI data as an assessment tool. Specific 
strengths of the UI data system are 
highlighted in Chapter 3. 
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Chapter 4 focuses on the development of 
the database of matched JTPA and UI 
records from the 11 states. We identify 
specific data elements included in all of the 
matched files and discuss the problems 
associated with developing the database. 

Chapters 5-7 report on applications of the 
matched state files. In Chapter 5, we 
compare post-program data based on UI 



wage records to data from the 13- week 
follow-up survey. In Chapter 6, we examine 
pre- and post-earnings patterns of client 
subgroups defined on the basis of the 
program services they received. Chapter 7 
considers the role of the wage record data In 
the context of the JTPA performance 
standards system. The report conduct 
with a summary of findings and tneir 
implications for the JTPA system. 
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Chapter Endnotes 

1. Program Year 1986 extended from July 1, 1986, to June 30, 1987. 

2. At this writing, the project is in its second phase involving PY87 data and four additional 
states. The latter are Georgia, Kentucky, Maryland, and Texas. A third phase will add PY88 
data and more states. nUnois remains as the project's clearinghouse, a role which continues 
to be executed through the Center for Governmental Studies at Northern Illinois University. 
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An Overview of Issues Concerning the 
Use of UI Wage-Record Data 



For over 50 years, unemployment 
compensation has provided America's 
workers with financial assistance during 
temporary periods of involuntary job loss. 
The business community, whose tax 
contributions help support the 
unemployment insurance system, has also 
benefited from this program. Research 
indicates that the UI system reduces 
recessionary pressures in times of local and 
national economic downturns. The bulk of 
unemployment benefits immediately 
returns to the local economy through 
purchases of retail goods, services, and 
nondeferrable expenses. 

This chapter gives a brief background of 
the UI system and describes its current 
status. We examine recent federal 
legislation that has laid the groundwork for 
the increased use of UI data in the 
management and evaluation of federally 
funded programs. In addition, we explore 
the issues of coverage, accuracy, timeliness, 
and confidentially to determine whether 
they represent serious barriers to the use of 
UI information as an assessment tool. 

Background of the UI 
System 

A national UI system in the United States 
can be traced to the Federal Unemployment 
Tax Act (FUTA) of 1937 (U.S. Congress, 
January 25, 1990, p.3; US. Department of 



Labor, January 3, 1988). This act established 
a federal tax on employers of 3% of a 
specified wage base. A 2.7% credit against 
this 3% gross rate was offered to each state 
that complied v/ith federal standards for the 
administration of a state unemployment 
compensation program* Within a few years^ 
all of the states had established eligibility for 
this 2.7% credit. The federal standards 
consisted of the following requirements: 

• Covered employers had to report 
total covered earnings each quarter 
so that the applicable tax could be 
conr.puted and assessed. 

• Covered employers had to be able to 
document each employee's 
earnings, so that eligibility to receive 
unemployment insurance benefits 
could be determined if it became 
necessary to do so. 

All covered employers in each of the states 
were required to comply with the first 
standard by reporting to the state 
employment security agency within a 
specified period of time. 

The states adopted one of two procedures 
to comply with the second federal standard. 
Many states required covered employers to 
report the earnings of individual employees 
each quarter This practice became known 
as wage-reporting. The remaining states 
adopted a second approach, called 
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wage-request. This approach allowed 
employers to defer reporting individual 
earnings until the information was 
requested to process an unemployment 
insurance claim. The Deficit Reduction Act 
of 1984 diminished the attractiveness of 
wage-request option by requiring all states 
to collect information on individual 
earnings each quarter. All but three of the 
states have adopted the wage-reporting 
procedure. Massachusetts and New York 
comply with this mandate through a state 
authority other than the state employment 
security agency. Michigan collects the 
information through its employment 
security agency but still maintains the 
wage-request procedure for administrative 
purposes. 

Congress has been increasingly willing *o 
require the use of state wage-record data for 
a variety of purposes. This tendency 
concerns the administrators of state 
unemployment insurance programs, who 
are responsible for protecting theanonymity 
of employees and employers. Some 
administrators are concerned that 
secondary uses of the information will 
interfere with the primary purpose for 
collecting the data-administration of the 
state unemployment compensation 
program. They are also concerned that the 
state employment security agencies may 
ultimately bear the costs of providing the 
information to external parties without 
reimbursement, a serious problem in view 
of shrinking budgets. They also point out 
that a clear understanding of the nuances of 
the data is essential to reliable applications. 
In 1988, the Employment and Training 
Administration of the U.S. Department of 
Labor created a task force to explore the 
issues involved in the use of UI wage records 
by external parties and the implications of 
that use for the UI program. 



The federal statutory language that is most 
directly relevant to this report appears in 
Section 106(b)(3)(C) of the Job Training 
Partnership Act, which states that 
performance standards will include 
provisions governing "cost effective 
methods for obtaining such [performance 
standards] data as is necessary to carry out 
this section, which, notwithstanding any 
other provision of law, may include access 
to earnings records . . . , state employment 
security records. Federal Insurance 
Contributions Act records, State Aid to 
Families with Dependent Children records, 
statistical sampling technique? and similar 
records or measures" (empha; is added). At 
least with respect to the eleven states that 
participated in the first phase of this project, 
an important feature of the project is to 
demonstrate that state laws and 
administrative regulations need not act as a 
barrier to responsible use of the wage-record 
data in full compliance with both the spirit 
and the case law of state privacy 
requirements. 

Coverage provisions, both with respect to 
employer liability to pay taxes and 
employee rights tc accrue eligibility, are 
state prerogatives. However, federal 
requirements associated with approval of 
the current 5.4% credit against the 6.2% 
gross tax rate have created a substantial 
amount of de facto uniformity among the 
states (U.S. Congress, January 25, 1990; U.S. 
Department of Labor, January 3, 1988; U.S. 
Congress, February 22, 1990). 

Up to 1956, the federal requirement 
applied only to employers of eight or more 
workers for one or more days in at least 20 
weeks during a calendar year. The 
minimum number of paid employees was 
reduced to four in 1956, and then to one in 
1972. Coverage now includes cases in which 
at least $1,500 in wages has been paid in any 
quarter during the current or preceding 
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calendaryear,excludingcertain agricultural 
and domestic service cases (see Appendix A 
for state-by-state coverage provisions). 

The practical importance of these coverage 
definitions is that one employee earning 
$ 11 6 a week for 1 3 consecutive weeks makes 
an employer liable for payment of the tax. 
Almost al! jobs that might provide 
employment opportunities for JTPA 
participants are likely to satisfy this 
minimum criterion for coverage. However, 
there are exceptions. Federal provision with 
respect to agricultural employment is a 
quarteriy payroll of $20,000 or 10 or more 
paid employees in each of 20 weeks during 
the current or preceding calendar year. 
Railroad employees are covered under 
separate legislation. Independent 
contractors whose service is performed 
outside the employer's place of business and 
who are free from control over performance 
are not covered in most cases. This group 
includes many real estate sales 
representatives and some other 
commissioned personnel. Self-employed 
individuals are not liable. Military 
personnel and federal government 
employees are not covered. Some part-time 
volunteers for nonprofit organizations, 
some employees of religious orders, and 
some students employed by their schools 
are not covered. Despite these exemptions, 
it is estimated that over 90% ' workers in 
the U.S. economy are in jobs co . ered by the 
UI system. 

Many different exemptions from coverage 
and tax liability have been granted in the 53 
political juris iictions. Some observers 
therefore argue that lack of uniform 
coverage makes wage records unreliable for 
nationwide longitudinal tracking. 
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Recent federal initiatives promise to 
improve the quality of the wage- record data 
for JTPA tracking purposes. The Bureau of 
Labor Statistics of the U.S. Department of 
Labor is creating a business establishment 
list, which will identify employment by site. 
Currently, employers who operate more 
than one facility in a state often have the 
option of reporting employment for each 
unit separately or for all of them combined. 
This has made it difficult to trace 
employment dynamics by size of firm or 
substate area. The Bureau of Labor Statistics 
is working with the state employment 
security agencies to convert to universal 
establishment reporting within the next few 
years. Current plans will require employers 
to distribute the total number of employees 
among production sites, but not to identify 
individual employees by location of 
employment. 

The Unemployment Insurance Service has 
begun a quality control program to correct 
weaknesses in the tax collection process. 
The rapid growth of small businesses, 
particularly in the services sector, has 
created concern about possible 
noncompliance problems (Burgess, 
February 21, 1990; Burgess, February 22, 
1990). 

Congress has enacted several requirements 
for the use of wage records. During recent 
sessions, for example. Congress has directed 
that UI wage records be used to verify 
income and to locate parents responsible for 
child support. These mandates have 
required the Unemployment Insurance 
Service and the state employment security 
agencies to reexamine their policies and 
procedures Tor responding to such 
requirements. 
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General Issues 
Concerning ttie Use of UI 
Wage-Record Data 

FoiiT topics arise in most discussions about 
the potential use of UI wage-record 
Information for longitudinal tracking 
purposes: coverage, acciiracy, timeliness, 
and confidentiality. We examine each of 
these topics and assess whether the issues 
they raise pose a serious barrier to using Ui 
data as an evaluation tool. 

COVERAGE 

Five types of employers are subject to 
coverage: (1) for-profit businesses, (2) 
nonprofit organizations, (3) nonfederal 
government entities, (4) employers of 
household employees, and (5) agilcultural 
enterprises. The conditions under which 
employers are liable depend on the type of 
employer they are. For-profit, 
private-sector businesses become liable 
based on four criteria: (1) paying a specified 
total payroll during a designated time 
period; (2) having one or more paid 
employees during a specified number of 
weeks in a defined time period; (3) acquiring 
and continuing another party's business; or 
(4) being liable under the FUTA and 
employing one or more paid workers in a 
Ltate. 

Ncnfedera' government entities are liable 
regardless of their payroll or the number of 
weeks their employee worked. Only very 
small nonprofit agencies are exempt from 
tax liability. Agricultural employers are 
liable if they employ 10 or more workers in 
20 different weeks during a year or if they 
pay $20,000 in cash wages during any 
calendar quarter. Both conditions apply to 
cumulative employment in all states. 
Employers of paid household help are liable 



if they pay at least $1,000 in cash wages 
during any quarter. The specific payroll and 
employment thresholds are not uniform 
among the states, but the FUTA standards 
must be met to remain eligible for the federal 
tax credit. 

Each state maintains a record of covered 
employment in that state only. All of the 
states use social security numbers to identify 
individuals. It is therefore possible, in 
principle, to determine whether a particular 
person appears in any state's adnunistrative 
records. There are many precedents for 
realizing this Potential, including the 
exchange of wage-record information 
between Illinois and Missouri during this 
project. 

Federal civilian and military employees, 
U.S. postal service workers, railroad 
employees, and most independent 
contractors and employees of religious 
organizations are not covered. However, 
the Education and Training Placement 
Information Program in Florida has 
established pioneering relationships with 
the federalOffice of Personnel Management, 
postal service administrators, and the 
Department of Defense to secure 
employment and earnings information on 
these sectors, too. Florida's success can be 
expected to result in other requests to these 
agencies. A high priority should therefore 
be given to establishing uniform policies for 
responding to such requests. In the absence 
of such policies, it is iiievitable that the cost 
and logistical disruption of multiple 
requests for data will result in a blanket 
refusal to cooperate. The design of these 
policies must include reliable estimates of 
the cost of specified types of cooperation. 
Self-employed individuals are not covered. 
Individuals in this categor ■ represent an 
important challenge in creating 
comprehensive longitudinal files. 
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Reportable compensation can includt 
regular cash payments, bonuses, 
commissions, tips, vacation pay, severance 
allowances, and the cash value of other 
compensation (e.g., food and shelter). There 
is no ceiling on the amount that must be 
reported, unlike the Social Security 
reporting requirement. 

State exemptions from tax liability result 
from successful lobbying in individual state 
legislatures, and many different exemptions 
have been granted in the 53 political 
jurisdictions. Some observers therefore 
argue that lack of uniform coverage n^kes 
wage records unrehable for longitudinal 
tracking. In very few circumstances, 
however, is this viewpoint compelling, and 
in such cases complementary data collection 
procedures can easily be adopted. Despite 
these exemptions, it is estimated that over 90 
percent of workers in the U.S. economy are 
in jobs covered by the UI system. 

The practical importance of these coverage 
definitions is that having one employee 
earning $116 a week for 13 consecutive 
weeks makes most employers liable for 
payment of the tax. Almost all jobs that 
might provide employment opportunities 
for JTPA participants are likely to satisfy this 
minimum criterion for coverage. 

ACCURACY 

Four areas ra^se concern about accuracy: 
(1) the identification of employers for the 
purpose of assessing job retention; (2) 
employers' reports of a covered employee's 
presence; (3) employers' reports of the 
money earnings paid to an employee; and 
(4) employerr/ reports of nonwage 
compensation. The contribution and wage 
report, which every covered employer 
submits in paper or electronic form each 
quarter, is the source for each of the four data 
items. 



The employee's social security number is 
the only way to link reported earnings to a 
specific individual. Reporting errors sever 
the connection between earnings and the 
person to whom they should be credited 
(U.S. Department of Health and Human 
Services, January, 1990). Algorithms have 
been developed to detect invalid number 
sequences and to reveal transposed digits in 
sequential quarterly wage reports. 
Employees who attempt to avoid taxes by 
using multiple or false social security 
numbers create a different type of problem. 

The employer identification number 
typically contains 14 digits, although each 
state sets its own reporting standard, i jt 
illustrative purposes here, a number may 
contain six digits that represent a unique 
employer account code. The seventh digit 
may be changed when ownership is 
transferred to another party so this becomes 
a flag to look for such changes. The next 
three digits might give the federal 
information processing system (FIPS) code 
that identifies the location of the reporting 
unit, with exceptions that are described in 
the next paragraph. The final four digits 
may represent the standard industrial 
classification (SIC) code for this business. 

Employers conducting business in more 
than one location in a single state may 
request a unique account code for each 
establishment, but they are not required to 
do so. When they do not make this request, 
the FIPS code is an ir\accurate identifier of 
the actual work site. The reporting practices 
affect different numbers of covered workers 
in a particular state because of the unique 
size and ownership characteristics in each. 
The business establishment list of the 
Bureau of Labor Statistics can be expected to 
create more uniformity in reporting 
practices among the states, but it will still not 
be possible to link individuals to their work 
sites. The specific features of 
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multiestablishment reporting practices 
across the states should be documented. 

A four-digit SIC code is assigned to each 
reporting unit when a business first 
establishes tax liability, and the code is 
subject to review every three years 
thereafter. The accuracy of this code 
depends upon the information that the 
business initially gives to the state 
employment security agency, the 
interpretation of the information by the 
agency employee who assigns the code, and 
the repetition of these two activities during 
each review cycle. Individual states have 
attempted to assign different SIC codes to 
major subsidiaries of some major reporting 
units, but no uniform practice of this type 
exists. 

The six-digit employer identification 
number is specific to each state. Use of an 
employer's federal identification number 
would facilitate interstate tracking of 
covered earners. A reporting unit may be 
identified with a location other than the 
actual worksite. A federal common 
paymaster plan is used in unusual 
circumstances such as longshore 
assignments, some construction work, and 
the growing temporary help sector. States 
also permit joint employer accounts to be 
established in order to determine taxes. 
Each of these practices increases the 
difficulty of assigning an individual to a 
work site. 

The accuracy of the reporting of money 
wages is unknown. However, relatively few 
corrections occur in the routine processing 
of individual unemployment insurance 
claims. In addition electronic payroll 
processing is increasing, electronic 
cross-matching capabilities are expanding, 
and new revenue quahty control practices 
have been introduced. Thus, there is reason 
to think that the accuracy of UI data is higher 



than that of self-rep>orted sources of earnings 
information. Intentional underreporting of 
wages constitutes fraud, which is subject to 
sanctions. Unintentional misreporting is 
subject to penalty payments. Intentioital 
reporting errors introduce a downward 
bias, since overreporting would create a 
higher tax liability. Unintentional 
misreporting should not re\'eal any pattern. 

The incidence of off-the-books payments, 
tips, and nonwage forms of compensation 
varies highly across occupations and 
industries. The food and beverage, private 
household, and professional seA^ice sectors 
are particularly vulnerable to intentional 
underreporting. 

This examination of the combined issues of 
coverage and accuracy supports the 
following conclusions: 

• Most jobs that would be considered 
appropriate targets for the 
placement of JTPA terminees are 
found in the covered sector. 

• Virtually all of these jobs offer 
money wage, commission, and 
bonus forms of compensation-all of 
which are required to be reported. 

• The exceptions to this general rule 
are concentrated in a few 
occupations and industries, 

TIMELINESS 

Employers are required to submit their 
contribution and wage report within one 
month of the end of each calendar quarter. 
They can request extensions, but they are 
liable to penalty and interest charges for late 
payment of taxes. 

State employment security agencies mail 
contribution and wage report forms to liable 
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employers near the end of each quarter, 
except when arrangements have been made 
for electronic reporting. The accuracy of the 
mailing depends upon the cooperation of 
other state agencies as they become aware of 
the formation of new businesses. This 
cooperation is a function of each state's 
commitment of resources to cross-matching 
and enforcement activities. 

The size of businesses affects the timeliness 
of reporting. Small businesses are less likely 
than larger ones to report electronically or to 
have specialized personnel routinely 
responsible for completing reports. Thus, 
small businesses would be expected to have 
a higher incidence of late reporting and a 
higher inaccuracy rate (Burgess, February 
22, 1990X 

Currently, wage-record data are available 
six to nine months after they are collected. 
The information employers are required to 
submit within 30 days of the end of a quarter 
must then be entered into the administrative 
data system of the state employment 
security agency. For instance, January data 
reported in April would normally be 
expected to be av£ulable in July. The data 
must be available by then so that the agency 
can use it to administer the unemployment 
compensation program. The speed of a state 
agenc/s response to third-party requests 
for access to administrative data typically 
depends on other data^processing priorities, 
what is requested, and whether cost 
recovery is offered. Requests for selected 
infomiation iten\s and those that require file 
merging impose a greater burden on the 
agency. For example^ if information is 
sought fr. m the agency's employer file 
about the total number of employees or total 
payroll amount, then the wage records and 
employer files have to be merged. Such 
requests can be expected to take longer. 



CONFIDENTIALITY 

Each state unemployment compensation 
statute contains a confidentiality provision 
that protects the anonymity of employees 
and employers. These stipulations were 
introduced when the programs were first 
established in the 1930s. The maturation of 
data processing capabilities, combined with 
widespread interdependences among 
private- and public-sector agents, have 
created a concern that disinterested third 
parties will violate these confidentiality 
standards. 

Those who seek to limit access to the data 
cite examples of the unauthorized use of the 
data. Those who advocate expanded access 
for responsible parties point to the historical 
record of authorized use in full compliance 
with state confidentiality requirements as 
evidence of successful cooperation. 

Many ways to encrypt wage records exist 
so that individual social security numbers 
and employer identification numbers 
cannot be read. These methods permit 
recurring mergers to be performed to 
lengthen a longitudinal record. The major 
stumbling block to date has been a limited 
sense of the urgency of establishing 
confidentiality standards and practices of at 
least minimal uniformity. This limited 
interest was a natural response to a small 
number of requests for access. Now, v^th 
congressional mandates and requests to use 
the data multiplying, the Department of 
Labor has begun to investigate the issue. It 
will be easy to compile documentation of 
security procedures used by other public 
agencies, e.g , the Internal Revenue Service, 
the Social Security Administration, the 
Bureau of Labor Statistics, the Department 
of Defense, and the National Institutes of 
Health. 
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Abuse of confidentiality is not inevitable. 
Awareness of the privacy standards and of 
the sanctions that will be imposed for 
violating them will control most of the 
vulnerability of the data. Common sense 
caution in the handUng of the records will 
eliminate most of the risk that remains. It is 
reasonable to assume that the heightened 
awareness of the issue will lead to tighter 
security. 

The continuous wage and benefit history 
program conducted by the Unemployment 
Insurance Service from 1977 through 1983, 
which included 14 states, and this project, 
which in its first phase includes 1 1 states, are 
just two of many examples of state 
cooperation . Additional examples include 
voluntary release of the wage records to 
universities, to non profit and for-profit 
research organizations, and to other state 
agencies. 



Summaiy 



This chapter examined the issues of 
coverage, accuracy, timeliness, and 
confidentiality. Well over 90 percent of total 
employment is covered by the UI system. 
The complementary interests of employees, 
reporting employers, and the state 
employment security agency foster 
accuracy. Advances in data processing 
technology continue to make data available 
more quickly. Confidentiality in the use of 
the data can be protected by providing 
access only to responsible parties who use 
appropriate technologies and practices to 
protect the data and who understand the 
sanctions that will be imposed for misuse 
(Middlebrooks and Stevens, forthcoming). 

For the purposes of tracking JTPA 
terminees, the technical issue that remains is 
the generation of reliable cost estimates for 
acquiring the data. This pilot project has 
taken a fir ,tep toward the assembling of 
reliable cost data, including interstate 
exchanges of information. These cost 
estimates are discussed in Chapter 5. 
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Principal Advantages of UI Wage Record Data 
for Program Analyses and Evaluation 



In this chapter we discuss the ways in 
which wage-record data from the 
Unemployment Insurance system can 
enhance the scope and quality of the 
information currently available to policy 
analysts and program decisionmakers. The 
UI system is a rich yet underused source of 
information. The richness of the UI files 
results from the variety of ways the 
information can be analyzed, not the 
number of items that are reported. This 
flexibility offers the opportunity to view the 
experiences of JTPA clients from a number 
of different perspectives, which, in turn, can 
assist decisionmakers in formulating new 
policies and evaluating existing policies. 

The Flexibility of UI wage 
Records as a Data Sotirce 

One of the most attractive features of UI 
wage records is the opportunity they afford 
to tailor analyses to the specific questions of 
decisionmakers* For example, questions 
concerning the post-program earnings of 
clients who received different program 
interventions can be explored by examining 
the earnings patterns of participants 
grouped by the types of interventions they 
received during the program. On the other 
hand, questions concerning differences 
among SDAs with regard to the 
pre-program employment experiences of 
their clients require that these experiences be 
summarized for individual SDAs. 



The ability of the UI data to fulfill these and 
other informational needs is a consequence 
of the way the data are reported. Each 
quarterly record contains information from 
a single employer for a single employee. 
This highly disaggregated level of reporting 
allows analysts using the data to design their 
studies with a great deal of flexibility. For 
example, when JTPA data are merged with 
UI data, it is possible to examine long-term 
pre- and post-program earnings of JTPA 
clients by (1 > the characteristics of the clients/ 
(2) the types of services they received, (3) 
geography, e.g., county, SDA, state, (4) 
industry of previous employment, (5) 
industry of placement, or (6) any 
combination of the above. 

Although the ability to summarize UI data 
across various dimensions is an important 
feature, the greatest value of this database 
lies in the opportunity it provides to focus 
analyses at the individual level^ Analyses at 
the individual level provide insights into the 
employment and earnings experiences of 
JTPA participants that cannot be obtained 
from summary data. For example, such 
analyses can uncover the factors that 
promote employment retention among 
former JTPA clients. It is even possible to 
determine whether these factors are 
different across subgroups of the JTPA 
population, e.g., among males and females, 
welfare and nonwelfare recipients, etc. 
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Capacity to Provide 
Longitudinal Data 

The examples above point to the potential 
for using UI data to examine the long-term 
employment and earnings experiences of 
JTPA clients. Currently, much of what is 
known about their post-program 
experiences is limited to the information 
collected through DOL's mandatory 
13- week post-program survey. Although 
this survey can provide a valuable range of 
information not contained in the UI records, 
it offers a short-term snapshot of the 
post-program experiences of clients. Many 
observers feel that the information is 
inadequate to judge the long-term success of 
program participants. This perspective is 
shared by the Senate's Labor and Human 
Resources Committee, which has proposed 
an amendment to the JTPA legislation 
prescribing a performance standard based 
on employment retention of six months. 

Quarterly UI wage records offer the 
opportunity to examine the post-program 
experiences of clients far beyond the 
six-month period proposed by this 
amendment. As additional quarters of UI 
data become available, they can supplement 
previously archived information. This will 
permit the tracking of post-program 
employment and earnings patterns over an 
expanding period of time. 

Quarterly UI wage record data can also be 
used to obtain information concerning the 
pre-program employment and earnings of 
clients. Although the availability of this 
information is uneven because some states 
have only recently become wage reporting, 
all states are expected to maintain files 
containing at least five quarters of UI wage 
data in the future. If UI files are accessed 
when clients enter the JTPA program, at 



least one full year of pre-program data will 
be obtained. Many states, however, 
maintain archives that will permit access to 
much older records. For example, all the 
states participating in the current project 
were able to gain access to eight quarters of 
UI wage records, and one state provided 
twenty quarters of UI wage data for each 
client. 

Wage-record data from the 
Unemployment Insurance system for 
pre-program quarters will provide a much 
needed supplement to the sketchy 
information currently available on the 
pre-program employment experiences of 
JTPA clients. For example, the only 
nationally reported data item related to the 
pre-program employment experiences of 
JTPA participants is the number of clients 
who were employed fewer than 15 weeks of 
the 26 weeks prior to enrollment. 
Pre-program wage information is even 
scarcer The average pre*program hourly 
wage is reported only for Title III terminees. 
The use of UI wage records can substantially 
expand this information base without 
imposing an additional burden of collecting 
data on local program operators. 

Access to pre* and post-program UI wage 
records will permit detailed longitudinal 
analyses of the employment and earnings of 
JTPA clients. For example, it will be possible 
to determine the percentage of former JTPA 
participants who remained employed at 
various post-program time intervals. More 
focused analyses can reveal those factors 
associated with long-term employment 
retention. In addition, the employment 
histories of clients can be developed from 
pre-program UI wage records. These 
analyses will be useful in identifying 
subgroups of JTPA clients who need more 
intensive services. 
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The availability of employer identifiers in 
the UI wage-record data will further expand 
the scope of analysis. One analytical option 
would be to examine the issue of job 
retention among former JTPA clients. Here, 
the focus is not whether fonner clients 
remain employed but how long they renuiin 
on the job with the same employer. For 
example, it would be possible to know how 
many quarters JTPA chents remain with 
their employer at placement. Furthermore, 
it may be possible to provide estimates of the 
number of JTPA clients who work for more 
than one employer. 

Resource for Net Impact 
Studies 



UI wage records also hold great promise 
for studies attempting to assess the net 
impact of JTPA and other employment and 
training programs. Employment and 
training programs are intended to have 
long-term impacts on the lives of 
participants. Net impact studies are 
designed to assess these impacts by isolating 
the unique contribution of program 
participation to long-term employment and 
earnings. This is accomplished by 
comparing the labor market experiences of 
participants with the experiences of a group 
of nonparticipants eligible for the program. 
The control or comparison group is carefully 
constructed so that their labor market 
experiences approximate what the 
experiences of participants would have been 
if they had not participated in the program. 
The difference between the actual 
experiences of participants and their 
hypothetical experiences is used to estimate 
the net impact of the program. 

Regardless of whether experimental or 
quasi-experimental designs are used to 
estimate the hypothetical experiences of 



program participants, UI wage records can 
play a key role in net impact studies by 
providing the information necessary to 
develop long-term measures of 
employment and earnings. Since UI data 
can be obtained for both participants and 
nonparticipants, measures developed from 
this database will be consistently defined 
across the two groups and over time, 

A particular advantage of using UI wage 
records to construct employment and 
earnings measures is the ability to examine 
net impacts at various time intervals 
following program participation. As a 
result, it will be possible to determine 
whether the impacts of program 
participation identified for one time period 
tend to increase, decrease, or remain 
constant over time. 

Comparison with 
Alternative Data Soxirces 

SOCIAL SECURITY 
ADMINISTRATION RECORDS 

Among administrative data sources, the 
earnings files maintained by the Social 
Security Administration (SSA) pose the 
most serious challenge to UI wage-record 
files as a source of longitudinal data. 
However, there are serious drawbacks to the 
use of SSA files. Since SSA records report 
only annual earnings, analyses of the 
post-program trends in employment and 
earnings must focus on the year-to-year 
changes in the measures. Furthermore, 
since earnings above the taxable maximum 
are not reported, statistics generated from 
SSA data can be biased. 

The most critical weakness of the SSA 
earnings files is the substantial delay in 
obtaining the data. Typically the data are 
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three to four years old before they are 
released in a form that analysts can use. Even 
then, a considerable amount of time is 
required to compile and analyze the data 
before evaluation results can be published. 
This time delay significantly reduces the 
value of SSA information to decisionmakers. 

SURVEY-BASED DATA 

The alternative to using administrative 
records for developing longitudinal 
databases is to rely on data gathered in panel 
surveys. In a panel design, the same 
individuals are interviewed at two or more 
different times. Each time individuals are 
interviewed/ their responses are merged 
with information gathered from previous 
interviews. The result is a longitudinal 
database recording changes in the 
employment and earnings status of panel 
members. 

Although panel surveys can provide a 
wide range of information not reported to 
the UI system, they are subject to a number 
of biases. Information collected m panel 
surveys rests solely on the memory of panel 
members, and the recollection of events 
decreases over time. Moreover, memory 
acts selectively. Some aspects of previous 
labor force experience may be accurately 
recalled, others lost, and still others, 
distorted. Thus, employment and earnings 
histories constructed from information 
obtained in a panel survey may be 
incomplete and misleading. 

Problems associated with low response 
rates have a more general impact. A panel 
design requires periodic surveys of the same 
population. Since gaps in the data are 
unacceptable in many types of longitudinal 
analyses, high response rates must be 
achieved each time the panel is surveyed. If 
panel members fail to respond to one of the 



surveys, whatever information they may 
have provided previously declines in value. 
More important than the reduction in the 
number of cases is the fact that high attrition 
rates can destroy the integrity of the 
longitudinal data. The reasons for 
nonresponse are seldom random. The 
nujor source of attrition is the inability to 
contact panel members for follow-up 
interviews. Many of the reasons for these 
failures can be linked to the employment 
and earnings status of the f>anel member, 
e.g., a disconnected phone number may 
indicate a loss ot employment. This implies 
that a high rate of attrition will change the 
composition of the panel so that it no longer 
represents the target population. 

If the experience collecting JTPA 
post-program data can be used as a guide, it 
is extremely doubtful whether attrition from 
the panel can be sufficiently controlled to 
permit the development of a quality 
longitudinal database. Many SDAs are 
having difficulty in maintaining the 70% 
response rate required by DOL's 
thirteenth-week post-program survey.^ A 
one-year follow-up survey of JTPA 
participants will likely produce much lower 
response rates, possibly below 50%, unless 
substantial amounts of time and resources 
are devoted to administering the survey. In 
addition, a one year follow-up is likely to 
produce larger differences in response rates 
among status groups than the 
thirteenth- week post-program survey. 

Even if the quality of the data were not a 
concern, the cost of constructing a nationally 
representative database from survey data 
would be enormous. The additional 
number of cases required to draw 
comp>arisons across groups and programs 
would push costs to an astronomical level. 
Bishop (1989) considered the cost of 
implementing a survey-based impact study 
comparing four target groups and three 
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different program strategies. He estimated 
that the costs for data collection alone would 
be $120,000,000. 

The same database could be constructed 
from UI wage-record data for just a fraction 
of the cost. The present study attempted to 
access at least eight quarters of UI wage data 
for each of the 205,750 PY86 JTPA terminees 
in the combined 1 1 -state database. For their 
efforts, each state received $10,000, and the 
clearinghouse state received an additional 
$15,000, making the total cost of this study 
$125,000."' To consider the cost of gathering 
this information through a panel survey, 
assume a total of five interviews (an initial 
interview to collect all the pre-program 
information and four quarterly follow-up 
interviews) at a cost of $25 per interview.'* 
The total cost of collecting the data through 
survey methods would be $25,718,750. 
Viewed another way, if the funds allocated 
to this project were used to conduct a panel 
study, the $125,000 would permit the 
collection of longitudinal data for only 1,000 
of the 205,750 terminees. 



Summaiy 



This chapter has identified a number of 
strengths of UI wage-records as an 
assessment tool. The flexibility of this data 
source offers the opportunity to examine 
many previously unanswered questions. Of 
special interest is the ability to use UI 
wage-record data to conduct detailed 
analyses of the long-term employment and 
eami*^ gs patterns of JTPA participants. 

The data also has potential for use in 
assessing the net impact of JTPA and other 
employment and training programs. Since 
UI data are available for both program 
participants and a comparison group, the 
employment and earnings of the two groups 
can be defined and measured consistently. 
UI data can be obtained more quickly than 
other administrative data, and they are not 
subject to the biases and errors that affect 
survey data. 

Finally, conducting research with UI data 
is clearly cost-effective. Given the 
increasing scarcity of federal funds, this 
factor alone suggests that greater efforts 
should be made to take advantage of this 
database. 
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Chapter Endnotes 

1. The analysis of individual-level data does not jeopardize the confidentiality of the 
information. 

2. Training materials distributed at the PY90 Performance Standards Conference sponsored 
by the U.S. Department of Labor and the National Alliance of Business indicate that one-third 
of all SDAs v^ere unable to achieve the 70% response rate for adults in PY88. 

3. Although the start-up costs estimated by the states exceeded $10,000, all states estimated 
that their continuing costs would be below $10,000. 

4. The $25 per interview figure is based on a survey conducted in 1987 by the National 
Governors' Association. 
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Compiling the 11-State Database 



One of the primary goals of this project was 
to develop a database containing 
longitudinal UI wage-record data and JTPA 
program data. This chapter discusses the 
achievement of that goal. It is organized into 
four sections. The first section discusses the 
issues faced by the states in gaining access to 
the required data and outlines the 
procedures used to transfer the data files to 
the clearinghouse. The second section 
describes how the clearinghouse processed 
the state data and discusses the various 
checks performed to ensure the integrity of 
the data. The third section focuses on 
coverage issues and describes the data 
elements contained in the final database. 
The final section offers some observations 
concerning the problems encountered in 
develo. ing this database. 

Data Access Issues and 
Transfer Procedures 

The 11 states participating in this project 
organize their JTPA and UI programs 
differently. For example, in Illinois, 
separate agencies administer JTPA and UI; 
in Indiana, the functions are merged within 
the Department of Employment and 
Training Services. In Missouri, the 
relationship is more complex. Different 
agencies administer JTPA and UI, but the 
same data processing unit maintains the 
data for each. 



The states faced different challenges in 
obtaining access to the data for this project. 
In states where the same agency 
administered JTPA and UI programs, data 
access issues were a concern within the 
agency and could be addressed at that level. 
However, when different agencies 
administered the two programs, the issues 
were complicated by the need to develop 
interagency agreements and data sharing 
procedures between the agencies. 

The states also differed with regard to their 
experience in developing an interface 
between the JTPA and UI data systems. 
Some of the states had a long history of 
working with combined JTPA and UI data 
files. The necessary cooperative agreements 
and procedures were already in place and 
firmly established. On the other hand, this 
project offered some states their first 
opportunity to link the two data systems. In 
these states, interagency or 
interdepartmental agreements had to be 
forged or invoked for the first time in order 
to satisfy the data requirements of the 
project. 

One concern in accessing UI wage record 
data is the Issue of confidentiality. Some 
observers of this project felt that the states 
would experience difficulty in releasing the 
data because release could jeopardize the 
confidentiality of the information contained 
in the records. When requested, the 
National Commission for Employment 
Policy contacted the state's attorney 
general's office to provide assurances that 
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the data would not be released in a form that 
would compromise the anonymity of 
individuals or employers. As an added 
safeguard, some states encrypted client 
identifiers to further ensure the 
confidentiality of the information. 

Although the states were left to their own 
devices in gaining access to the data, the 
clearinghouse provided the states with 
procedures for transferring the data. Data 
were transmitted in three separate files. The 
JTPA file was defined so that it would 
contain all requested JTPA data items except 
those providing information about program 
activities. These latter data elements 
comprised the second activity file. The third 
file contained all the information extracted 
from the state's UI data system. The data 
items requested for each of the three files are 
listed in Appendix B. 

Previous experience of the clearinghouse 
staff with the JTPA management 
information systems (MIS) in various states 
indicated that the two-file strategy would be 
the most effective means for transferring the 
JTPA data. Most state systems employ a 
hierarchical data structure for storing JTPA 
data. This structure, which was assumed to 
be common among the participating states, 
requires that each activity in which a 
participant is involved be recorded as a 
separate data record. Thus, it is relatively 
easy to develop a separate data file 
containing one record for each activity. 
However, padding each participant's data 
record with activity information would 
require some intricate programming and 
substantially increase the amount of time 
necessary to produce and transfer the file. In 
addition, since the procedures developed to 
process activity data required that a separate 
file be developed, requesting that the 
participant record be padded with activity 
data seemed to be counterproductive. Thus, 



the two-file option for the JTPA data was 
adopted. 

The third state file, the UI file, contained 
quarterly UI data. This file was defined so 
that each data record would contain UI data 
for a single quarter. Since it is possible for a 
terminee to have more than one employer 
during a quarter, this file can contain more 
than one record per quarter for a participant. 
Thus, the file is considered to be 
employer-based. For each data file, the 
clearinghouse defined a data format that 
specified the locations for each requested 
data item, and strongly encouraged the 
states to use these formats in constructing 
their data files. These formats included a 
user-defined segment where states could 
provide supplemental data if they desired. 

The clearinghouse also provided the states 
with a suggested coding scheme for each of 
the data items, but we did not insist that 
states use these coding conventions. If a 
state chose to deviate from the scheme, it 
was instructed to document the differences 
and submit this documentation with its data 
files. 

A question that emerged from several 
states was whether to include essentially 
blank records on the UI file for quarters 
where a match did not occur. The 
clearinghouse left ihis decision to each state. 
It would have been desirable, however, for 
the states to include such records. They 
would have provided a direct means for 
assessing whether there had been an attempt 
to obtain a UI match for each participant in 
the JTPA file. At the present time, we have 
only indirect evidence of the completeness 
of the files in those states that chose not to 
include blank records. 
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Processing the State Files 

We used a number of procedures to insure 
the integrity of the state databases. For the 
JTPA data file, these checks focused on the 
post-program data. For example, a 
procedure was used to test whether all of the 
mandatory post-program data items 
contained valid entries when a 
post-program survey completion was 
recorded. When a mandatory item was 
missing, we changed the status of the record 
to indicate a noncompletion. Furthermore, 
if a noncompletion was indicated, we reset 
the values of all the post-program data items 
to "missing" regardless of whether valid 
data were present for one of these items. 
Although this procedure destroyed the 
information gained from partial 
completions of the survey, it imparted some 
degree of consistency to the state databases. 
It also corresponds to the way data are 
reported to the U.S. Department of Labor. 

The major issue concerning the use of the 
UI files was whether all the states used the 
same decision rules to identify pre- and 
post-Ul quarters. Therefore, we asked states 
to include a data item on the UI files that 
indexed each UI wage record to the program 
participation dates of the corresponding 
client. For example, if the state followed the 
suggested coding scheme, a code of TOl 
indicated that the record contained UI data 
corresponding to the client's first full 
quarter after temiination and a code of E02 
indicated that the record corresponded to 
the second full quarter prior to enrollment. 
Without this index, it would have been 
extremely difficult to determine whether the 
states were interpreting the UI data in the 
same way. 

We used a simple procedure to assess 
whether the states were defining pre-and 
post-program quarters consistently. First, 



the clearinghouse constructed its own 
quarter index using the dates of program 
enrollment and termination from the JTPA 
file and the starting date of the quarter from 
the UI file. We then compared this index to 
the index calculated by the state. If the two 
indexes did not match, we attempted to 
identify the source of the discrepancy and 
esolve the differences. This procedure 
revealed that, with one exception, all of the 
states had calculated the index correctly. 
We notified the remaining state so that it 
could correct the errors in its file. 

We used a procedure to indirectly assess 
the completeness of the UI files for those 
states that excluded nonmatched 
participants from the UI database. Using the 
merged JTPA and UI file we performed a 
check to determine whether an entered 
employment reported on the JTPA file could 
be verified with the UI data. Although we 
did not expect that all JTPA reported 
employment could be verified with the UI 
wage data (e.g., those clients who gained 
employment in other states), we expected 
the vast majority of these cases to be 
matched. If we found an abnormally low 
percentage of entered employments 
matched in a state, 'his would suggest that 
the UI file was inco^nplete. This procedure 
was instrumental in identifying an 
incomplete UI file in one state. We informed 
the state of this problem, and a new file was 
processed. 

The activity file required the most 
extensive processing. Since activities are 
coded differently in each state, the major 
task was to translate the state codes into a 
common coding scheme. The clearinghouse 
developed a coding scheme consisting of 
nine categories. The goal was to minimize 
the number of categories while retaining the 
basic distinctions between broad classes of 
activities. The scheme appears in Table 4.1. 
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Table 4.1 

Categories Used To Recode State Activity Data 

1. Occupational Classroom Training . Training provided in a classroom-like setting 
designed to promote the technical competence needed for a particular type of job. 
Examples include vocational skills training and pre-apprenticeship training. 

2. Remedial or Basic Educational Training . Training provided in a classroom-like 
setting designed to promote basic skills development. Examples include English As a 
Second Language programs and GED preparation. 

3' On-the-job Training . Interventions that provide skills training in a specific 
occupation in an actual work setting. 

4. Work Experience. Part-time or short-term subsidized employment designed to assist 
participants in entering or reentering the labor force or in enhancing their 
employability. Both limited and regular work experience are included. 

5. Employment and Training Services. Activities designed to assist participants in other 
JTPA-funded training and to enhance employment opportunities, facilitate movement 
into unsubsidized employment, or assist in retaining employment. Included are labor 
market orientation, job counseling, and introduction to the world of work. 

6. Job Search Assistance. Interventions that offer aid to participants in locating, 
applying for, or obtaining a job. Included are direct placement, job referral, and job 
development. 

7. Youth Programs. Activities focused on improving the employability of youth 
participants. Included are exemplary youth programs and pre-employment training. 

8- Other. Interventions tha. cannot be included in the categories described above. 

9. Holding . 



The clearinghouse recruited four 
volunteers to recode each stage's activities 
independently. After each person 
completed this process, the clearinghouse 
examined the results to identify any 
differences in the codes assigned to 
particular activities. Fortunately, only 
:ninor discrepancies appeared. In these 
cases, we assigned the code used by the 
majority to that state's activity. 



Once the activity data were recoded, the 
clearinghouse processed the data to produce 
a single variable for each participant that 
could be used to assess the sequence of 
activities in which the participant engaged. 
Through this activity sequence indicator, it 
is possible to determine the activities and the 
order of activities in which a participant was 
involved. 
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We constructed a number of other 
variables from this file. For example, we 
constructed a variable to signal wrhether a 
participant received support services, e.g., 
needs-based payments. We also 
constructed a set of variables to reflect the 
length of time participants v^ere enrolled in 
each of the nine categories of activities and 
the length of time they received support 
services. 

We performed a check on each state's 
activity file prior to processing it in order to 
ensure that at least one activity or support 
service was reported for each participant. 
Although the result of this test was positive, 
some problems continue to exist with the 
activity data. Compared to other 
information, activity data are more likely to 
be miscoded when entered in the state data 
systems. Approximately 3 percent of all 
activity records the clearinghouse received 
contained codes that were not found in the 
state documentation. The actual percentage 
of miscodes is probably higher given that 
some activities were likely to be miscoded 
with a valid state code. We will discuss the 
impact of this situation later in this chapter. 

The construction of the state databases 
proceeded slowly at first. Much of the initial 
effort was spent in developing computer 
programs and data checking routines that 
could be used, with minor modifications, to 
process the data from all states. Once these 
programs were written and debugged, it 
took from four to six hours to modify them 
to accommodate each additional set of state 
files. Depending on the size of these files, it 
takes from one to three days from the time a 
data tape is received to process a state's files 
and produce a working database. 

Initially, the clearinghouse took the 
position that a common file format was 
needed in order to reduce the degree of 
programming required to produce the 



various state databases. However, our 
experiences with states that did not provide 
the JTPA data in the suggested format 
indicate that programming time is not 
appreciably increased as long as adequate 
documentation is available. 

A more important determinant of 
programming time was the extent to which 
the sUtes adhered to the suggested coding 
scheme. The data files from states that opted 
to use their own coding scheme required 
more programming time than files from 
states using the suggested scheme. 
However, the amount of additional 
programming was not great enoujj^h to 
warrant limiting the flexibility of the states. 
In fact, this flexibility has enriched the 
database. For example, although the 
suggested coding scheme included the 
category "employed" for the employment 
status at enrollment, some states further 
identified participants as employed 
part-time or full-time. 

Once the three data files submitted by a 
state were processed, they were merged to 
produce a single, individual-level database 
for each state containing all the data items 
provided by the state. At this point, we 
constructed a set of data tables for each state 
and cross-checked against a corresponding 
set of tables produced by the state. If we 
found discrepancies between the two sets of 
tables, we attempted to identify the source 
of th?se differences. On the basis of this 
information, either the clearinghouse or the 
state made corrections. 

The final step involved producing the 
11-state database by linking the separate 
state databases. Given the efforts to ensure 
that the individual state databases were 
accurate and consistent, this step was a 
formality. 
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Coverage and Available 
Data Items 

The states were requested to submit 
enrollment and termination data for all PY86 
Title 11- A and Title ni adult terminees and all 
PY86 Title II-A youth terminees who were 
not in school. Only two of the states fell 
short of this goal. One state was unable to 
provide information on its PY86 Title in 
terminees because it had technical problems 
gaining access to all the requested data items 
for this client population. The other state's 
omission of Title U-A youth data, however, 
appears to have been an oversight. 

The remaining coverage issues centered on 
the availability of post-program survey 
data. The amount of post-program data 
available in a state depends on whether the 
state surveyed all terminees or a random 
sample of terminees. Most of the states 
employed sampling techniques; however, 
four states surveyed all PY86 adult Title II-A 
terminees. 

Two of the 11 states could not provide 
post-program data that were representative 
of all participants that terminated from their 
programs in PY86. In one state, the state MIS 
system was not modified in time to p>ermit 
the SDAs to enter post-program data in 
PY86. However, individual-level Title II-A 
post-program data were available for four of 
the SDAs in the state, and the complete state 
Title III sample was available as well. 

A different problem affected post-program 
coverage in another state. Post-program 
start-up problems resulted in the failure to 
collect information from participants who 
termJ*>ated from the program prior to 
January 1, 1987. However, once data 
collaboration began, the state surveyed 
terminees from all SDAs. With some minor 
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omissions, the states were able to provide 
the clearinghouse with all of the JTPA data 
items it requested. Four of the 11 states do 
not record the age of the youngest 
dependent child so they were unable to 
identify whether a terminee was a single 
parent with a child under six years of age. 
Also one state could not provide the county 
of residence. 

Several states chose to supplement the 
requested data items with additional 
information on their JTPA participants, with 
these states providing the clearinghouse 
with complete access to their MIS data. 
Another state provided an impressive array 
of additional data items collected through its 
post-program survey, including the welfare 
status of terminees at follow-up, the amount 
of the welfare grant they received at 
follow-up, and whether their current job 
provided health insurance. 

It was encouraging to find that many of the 
states went beyond the minimum 
requirement of four quarters pre-program 
and four quarters post-program data in 
developing their UI files. Five of the 11 
states submitted additional pre-program 
quarterly wage data, and six states 
submitted additional post-program wage 
data. Furthermore, five states provided the 
clearinghouse with UI data for the quarters 
in which their terminees participated in the 
program. 

Although all of the states were able to 
provide UI data containing employer 
identifiers, one state was unable to provide 
the employer's industry code. That state 
collects the industry code information, and 
it is available on the state's on-line UI 
database; however, it was absent from the 
archived UI tapes the state used to collect the 
historical data for this project, Only two 
states were able to provide UI data on the 
number of weeks worked in the quarter, and 
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one state provided the number of hours 
worked in the quarter 

To extend the range of information 
contained in the final databases, the 
clearinghouse constructed a file containing 
SDA-level econonuc factors, including: (1) 
area average wages in 1986, (2) area average 
earnings in wholesale and retail trade in 
1986, (3) unemployment rate for PY86, (4) 
population density in 1980, (5) percentage of 
families below poverty in 1979^ and (6) the 
1980 employee/resident worker ratio. The 
source for this data was the Department of 
Labor's technical assistance guide for the 
PY88 performance standards models. 

The clearinghouse also developed a 
county-level economic database for use in 
this project containing four economic 
indicators calculated for each county in the 
11 states. The indicators include: (1) area 
average wages in PY86, (2) area average 
earnings in wholesale and retail trade in 
PY86, (3) unemployment rate for CY86, and 
(4) the employment growth rate between 
CY85 and CY87, This file permits a more 
extensive examination of the effects of local 
economic conditions on performance 
indicators. It is especially important for 
Title III analyses since SDAs did not always 
administer Title ni services in PY86, 

Observations and 
Assessment 

When we began this project, we anticipated 
that most problems would emerge in 
working with the UI data files. As it turned 
out, processing of UI data was surprisingly 
easy. The major problem was the difficulty 
one state had in correctly identifying pre- 
and post-UI quarters. However, this state is 
distinguished by the relatively large number 



of programming errors it made during the 
course of the project. 

In contrast, we experienced a number of 
problems in processing the JTPA data. For 
example, in aeating the JTPA database for 
one state, we noted that a high percentage of 
cases contained invalid codes for a 
particular data item. An examination by the 
state revealed that it had substituted a 
different data item in place of the one we had 
requested. As a result, the state had to 
generate a new file and transfer it to the 
clearinghouse, 

A more serious problem resulted when a 
state misunderstood the data transfer 
conventions. We expected the JTPA file to 
have only one record per program 
ternunation. However, we found that the 
file submitted by one state had multiple 
records per termination. After some 
discussion, it became clear that the state 
mistakenly believed that the clearinghouse 
wanted one record per termination from 
each activity, rather than one record per 
tennination from the program . 

Normally, this misunderstanding would 
not pose a major problem provided that the 
state defined the other data items in the 
JTPA file correctly. However, this was not 
the case. Specifically, the state used the 
enrollment and termination dates for the 
activity in place of the enrollment and 
termination dates for the program. Since the 
latter items are the keys to identifying and 
defining pre- and post-program UI quarters, 
a substantial effort was needed to rectify this 
situation. 

These two examples illustrate the point 
that many problems could have been 
avoided by eliminating the need for states to 
pre-process the JTPA data prior to 
transferring it to the clearinghouse. In the 
first example, the excluded data item would 
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have been available if we had requested a 
full rather than an abridged data file. In the 
second example, the simple request that 
states submit all of the JTPA data would 
have removed the basis for the 
misunderstanding. Although it was not 
difficult to correct these and similar 
problems, the effort involved delayed the 
production of the final databases and 
decreased the time available to analyze the 
data. 

Unabridged JTPA files could have been 
useful for other reasons as well. For 
example, they would have reduced the data 
processing burden on the states. In fact, 
Oregon asked to submit an imabridged file 
to the clearinghouse because it estinnated 
that an abridged file would cost over four 
times as much as producing an unabridged 
file, $7,500 as opposed to $1 ,600. In addition, 
unabridged files would have increased the 
number of available data items, which 
would, in turn, have increased the richness 
of the final database. For example, the 
information collected by the states to 
determine Title III eligibility would have 
been a welcome addition to the database. 

Of all the data items contained in the final 
database, those constructed from program 
activity information were the most likely to 
be biased. We found a relatively high 
number of miscodes when we cor>structed 
the activity sequence indicator. When we 
found these invalid codes, we excluded the 
unidentified activity from the calculation of 
the indicator. For example, if a participant 
engaged in an activity that could not be 
identified, we ignored that activity and 
processed the remaining activity 
information as if that activity had not been 
received. This procedure affected relatively 
few individuals, but it did introduce a minor 
source of bias in the data. 



A more important source of bias in the 
program activity data resulted from the 
nonreporting of program services. We 
suspect that an unknown percentage of 
services are never reported on state data 
systems. For example, a participant served 
through a performance-based contract may 
be reported as engaging in only one activity, 
e.g., occupational classroom training, 
although that participant could have 
received a variety of additional services 
through the contractor, e.g., job search 
assistance. Since the program activity 
information contained in this database 
reflects only reported services, it is likely to 
undercount the actual level of services 
provided to JTPA participants. Thus, the 
measures should be interpreted with 
caution, especially when focusing on 
activities that can be considered as adjuncts 
to other services. 

The problems surrounding the use of 
program activity data are a consequence of 
the lack of attention these data receive 
compared to the other data items reported 
on state JTPA data systems. All states 
closely monitor the quality of participant 
characteristic and outcome data because this 
information is important to their 
performance management systems. 
Furthermore, since SDAs receive incentive 
funds on the basis of this information, they 
have a vested interest in assuring that the 
data they provide are complete and 
accurate. 

It is doubtful whether program activity 
data receive the same level of concern and 
attention. Instead, we suspect that the 
quality of such data is a direct hinction of the 
role this information plays in the state's 
performance management system. Since 
some states take an active interest in the 
services offered by their SDAs, and other 
states have a hands-off approach, it is likely 
that the quality of this information varies 
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from state to state. Until program activity 
information attains a prominent role in all 
state performance management systems, the 
consistency of the data is likely to suffer. 

Fortunately, there is some relief in sight, at 
least in the case of the Title in program. The 
Title ni reporting requirements for PY90 
direct substate grantees to report the 
percentage of participants receiving specific 
types of ]TPA services. It is hoped tluit this 
mandatory reporting will lend some degree 
of consistency to the data. 

Summaiy 



We expected to uncover problems in 
matching and merging data from different 
systems within a state, let alone trying to 
construct a unified database spanning 
several states. As this chapter has detailed, 
hov^ver, we had few problems and most 
were readily correctable. In many respects, 
it is surprising that the number of problp' s 
was not greater given that this was the Hn-^ 
attempt to develop a multistate database of 
this type. 

One technical issue that remairis to be 
addressed concerns the ability of the states 
to integrate UI wage-record data into their 
JTPA data management systems. The 
current project did not require the states to 
modify their JTPA data management 
systems to accept UI wage-record data. As 
a result, we suspect that most states 
developed and analyzed their databases 
outside the context of this system. A logical 



next step is for these states to incorporate UI 
data elements into their JTPA MIS database 
to allow client records to be updated 
periodically as additional quarters of UI 
data become available. This will provide 
immediate on-line access to current 
information for both the states ana SDAs. 

The difficulty of integrating UI wage 
record data into existing JTPA data 
management systems will vary from state to 
state. A state's level of difficulty will 
depend on the ability of its JTPA data system 
to accept and store information from 
external databases. Some states have 
always had this capacity, or they have 
developed it in resporwe to the need to 
incorporate post-program survey data 
collected by outside contractors. However, 
other states maintain systems that require all 
data to be entered manually into the 
database. It will be necessary to overcome 
this limitation to permit UI data to be 
electronically transferred into the system. 

Further analysis of the issues involved in 
incorporating UI wage-record data into 
existing JTPA data systems would be 
beyond the scope of this project. It is 
sufficient to note that any problems that may 
emerge in this process can be characterized 
as obstacles to overcome rather than 
insurmountable barriers. The fact that each 
state was able successfully to merge the 
information from JTPA client records with 
quarterly wage data from the UI system 
suggests that a fully integrated data system 
is a real option for the future. 
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UI Wage Record Data Versus 
Post-Program Survey Data 



When post-program measures were 
introduced to the JTPA performance 
standards system, contact with participants 
was the only way to collect data that had the 
potential to provide accurate and reliable 
post-program information across all states 
and SDAs. Although other sources were 
considered/ none were sufficient to meet the 
specific needs of the JTPA program. For 
example/ the possibility of using UI wage 
records as a data source was dismissed as an 
option because only 38 states required 
employers to report the earnings of 
employe* ^ 'n the absence of a UI claim. This 
drawback was essentially eliminated with 
the passage of the Deficit Reduction Act of 
1984/ which mandated that all states report 
wages by October 1988. With this new 
mandate in effect, the Commission 
recognized the need to re-evaluate the use of 
UI data for JTPA post-program assessment. 

In this chapter we compare UI wage-record 
data with the information obtained through 
the nnandatory post-program survey. We 
examine the post-program survey by 
describing some of the factors ' iiat can affect 
the quality of the survey data and provide a 
context for assessing the quality of the UI 
wage-record data. Then we analyze some of 
the differences between the two sources of 
data and discuss the relative quality of the 
information each provides. Finally, we 
compare the costs of obtaining 
post-program data through these two 



options based on the experiences of the 
states participating in this project. 

Post-Program Survey 



A key consideration in the design of any 
survey is the establishment of procedures to 
safeguard the reliability and accuracy of ':he 
information collected . Issues of data qu;x Jty 
are accentuated in the JTP> post-program 
survey because critical decisions are based 
on the collected information. In designing a 
system to collect post-program data based 
on participant contact/ the U.S. Department 
of Labor (DOL) faced a number of 
challenges to ensure that accurate estimates 
of post-program performance could be 
obtained for all SDAs. Since the actual 
surveys were to be conducted at the state or 
in some cases the SDA level, DOL had to 
develop specific guidelines to promote 
consistency in the data collection efforts and 
maximize the quality of the follow-up 
information. 

Each of the decisions DOL made in 
designing the system can be viewed as an 
attempt to improve the overall quality and 
consistency of the post-program data. For 
example, although DOL permits each state 
to determine whether to interview all 
terminees or use a more cost-efficient 
sample of terminees, those states selecting 
the sampling option must adhere to rigidly 
defined sampling procedures. Each sample 
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has to be randomly selected from the 
terminee pool and has to be large enough to 
provide accurate estimates of performance 
for all the post-program measures. In 
addition, DOL requires the information to 
be collected directly from the participant 
either by telephone or in person. Less 
reliable mail questionnaires are permitted 
only when the participant cannot be reached 
by telephone. 

Nonresponse bias, the most common 
problem affecting the accuracy of survey 
data, is difficult to identify and control In 
the context of the post-program survey, the 
concem war that terminees not responding 
to the survey might have substantially 
different post-program experiences from 
those of respondents. In such a case, 
estimates of SDA performance based on 
survey responses will not reflect the actual 
performance of the SDAs. 

The only way to reduce nonresponse bias 
is to minimize the number of nonresponses 
to the survey. For this reason, DOL requires 
minimum response rates. Since there is a 
special concem that terminees employed at 
termination will be easier to locate for the 
survey than those not employed, DOL 
requires a minimum response rate of 70% for 
both groups. If the difference between the 
response rates for the two groups is greater 
than five percentage pointS/ SDAs use an 
adjustment procedure that attempts to 
correct for this possible source of bias. 

There has been very little research into the 
extent to which nonresponse bias affects the 
post-program survey data because there is 
no adequate database to support the 
research. The major source of information 
on response rates available at the national 
level is the JTPA annual summary reports. 
These reports provide only the overall 
response rates achieved by each SDA for the 
three survey populations (Title II-A adult 



terminees. Title II-A adult welfare 
terminees, and Title III terminees)* At a 
minimum, identifying nonresponse bias 
would require access to the response rates of 
client subgroups suspected of having 
different post-program experiences. 

The database developed through the 
current project offers the first opportunity 
for a detailed multi-state analysis of the 
response rates to the post-program survey. 
Since it is an individual-level database, 
response rates for different client subgroups 
can be calculated and compared. More 
importantly, the database incorporates VI 
wage-record data, which can be used to 
determine whether the response rates for the 
various groups are associated with their 
post'program employment and earnings 
experiences. 

In the analyses that follow, we focus on the 
survey conducted to produce estimates of 
the post-program performance of Title II-A 
adult participants. Across the eleven states, 
42364 FY86 adult terminees were selected 
for post-program follow up, and 27,721 
responded to all three of the mandatory 
questions-an overall response rate of 65.1 
percent. 

We found that certain subgroups are more 
likely to respond to the post-program 
survey than others. The greatest difference 
's between the two subgroups defined on the 
basis of their employment status at 
termination from the program (see Table 
5.1). Participants employed at termination 
from the program had a response rate of 70.2 
percent, much higher than the 49.6 percent 
rate for those who were unemployed at 
termination. In addition, we found 
substantial differences in response rates 
across comparable sex, race, education, and 
public assistance subgroups. These results 
suggest that males, minorities, high school 
dropouts, and welfare recipients are all less 
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Table 5.1 

Post-Program Survey Response Rates by Client Subgroups 



Title II-A Adult Survey Population 



Characteristic 


Response Rate 


Sex 




Males 


62.0 


Females 


67.5 


Race 




White 


66.8 


Black 


61.5 


Hispanic 


60.7 


American Indian or Alaskan Native 


52.9 


Asi;3n or Pacific Islander 


65.3 


Educational Status 




High School Dropout 


60.8 


High School Graduate 


66.4 


Post High School Education 


67.5 


Welfare Status 




Non Recipient 


65.7 


AFDC Recipient 


63.7 


General Assistance Recipient 


49.3 


Refugee Assistance Recipient 


62.4 


Employment Status at Termination 




Employed 


70.2 


Unemployed 


49.6 



likely to respond to the post-program 
survey. 

The next step was to determine whether 
differences in response rates among the 
demographic subgroups were statistically 
significant after we controlled for the effects 
of employment status at termination. Since 
the demographic subgroups we examined 
tended to have different levels of 
employment at termination, this step is 
necessary to elinninate the possibility that 
subgroup differences in response rates are 



m »rely a product of the difference in 
response rate between the employed and 
unemployed. 

The results of the multivariate analysis 
indicate that the differences in response 
rates among the demographic groups 
remain even after we control for 
employment status at termination (see Table 
5.2). Although those employed at 
termination are still the most likely to 
respond to the survey among all the 
subgroups examined, the fact that the 
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Table 5.2 

Regression Resulte for a Model Predicting Response to the Post-Program Survey 



Title II-A Adult Survey Population 



regression VArwDle 


Coefficient 


Employed At Termination 


.204 


Male 


-.073 


Black 


-.041 


Hispanic 


-.043 


High School Dropout 


-.043 


Welfare Recipient 


-.023 


.0433 





Note: All variables significant at alpha =.001 
N = 24563 



regression coefficients for the demographic 
subgroups are statistically significant (alpha 
= .001) indicates that the differences in 
response rates found among these 
subgroups are not explained away by 
employment status at termination. 
Response rate patterns for the JTPA 
population are similar to those found in 
surveys of the general population. 

The differences in response rates among 
the demographic groups helps to explain 
some of the variation in response rates 
found across SDAs. Even in states where a 
centralized system ensures that the same 
procedures are followed and the same level 
of effort is expended to collect data for all 
SDAs, the response rates of the SDAs tend 
to vary. Our analysis suggests that these 
variations are in part a function of the 
different clientele the SDAs serve. For 
example, SDAs that serve a high percentage 
of males should be expected to obtain lower 
response rates than SDAs that do not. 



This analysis also indicates that any 
attempt to impose sanctions on states or 
SDAs for failing to achieve the minimum 
response rate would be ill-advised. The 
failure may not reflect the quality of data 
collection procedures; it may reflect the 
types of clients served. In some respects, a 
r^ .ver response rate may indicate that the 
SDA is attempting to serve the most 
hard-to-serve component of the population 
eligible for JTPA. 

Differences in response rates cannot be 
used as evidence of nonresponse bias in 
survey data. At best, they merely signal that 
such bias might exist. For example, the 
observation that males are less likely to 
respond to the survey than females does not 
indicate that the estimates of actual 
performance are biased. This conclusion 
can be drawn only when it is also 
demonstrated that males and females have 
different post-program experiences. If the 
experiences of the two groups are the same, 
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then the feet that more females respond to 
the survey will not affect the estimates of 
actual performance. However, if their 
experiences are different, then the estimates 
of actual perfomnance will be biased since 
the two groups are disproportionately 
represented among respondents to the 
survey. 

Nonresponse bias can exist even when 
there are no observed differences in the 
response rates for client subgroups. Just as 
different response rates do not prove the 
presence of bias, the same response rates do 
not prove its absence. Suppose that males 
and females were equally likely to respond 
to the survey and were therefore 
proportionately represented among the 
survey respondents. In order to disnuss the 
possibility of nonresporii>.- bias, it must be 
shown that the respondents for each of these 
two groups have post-program experiences 
that are typical of their respective 
populations. In other words, the 
post-program experiences of both groups of 
respondents must be the same as both 
groups of nonrespondents . If the 
experiences of either group of respondents 
differs from those of nonrespondents, then 
the estimates of actual performance based 
on the survey information will be biased. 

The most troubling aspect of nonresponse 
bias is that it can rarely be identified. The 
only way to ascertain whether nonresponse 
bias is a problem is to have information for 
nonrespondents on the very behaviors or 
experiences the survey is intended to 
measure. Of course, if such information 
existed, there would be no point to 
conducting the survey. As a result, survey 
researchers must attempt to correct for 
nonresponse bias without the information 
that would enable them to do so. In the 
al)sence of this information, the attempts to 
correct for nonresponse bi;»s '-an actually 
accentuate the biases that do exist, for these 



reasons, nonresponse bias has often been 
characterized as the weak link in survey 
research. 

The access to UI wage-record data on the 
post-program experiences of virtually all 
JTPA terminees, whether or not they 
respond to the post-program survey, 
provides a unique opportunity to exanune 
the issue of nonresponse bias in the 
post-program survey data. Although there 
are some coverage issues with regard to the 
UI data, their impact should be randomly 
distributed across the respondent and 
nonrespondent populations- In other 
words, we can expect each of these two 
groups to be equally affected by whatever 
coverage problems exist. Consequently, it is 
possible to use UI wage information for 
post-program quarters to determine 
whether respondents and noru-espondents 
have different post-program experiences 
and to examine the nature of these 
differences. 

Using UI data to compare the 
post-program experiences of respondents 
and nonrespondente suggests the potential 
for a high degree of nonresponse bias in the 
survey results. Although 70.7% of 
respondents were in Ul-covered 
employment during the first post-program 
quarter, only 61.1% of nonrespondents 
were. This implies that estimates of 
post-program employment derived solely 
from the employment experiences of survey 
respondents will overstate the actual level of 
employment in the terminee population. 
There is also evidence that nonresponse bias 
affects the post-program earnings measure. 
The UI wages of respondents who were 
employed during the first post-program 
quarter averaged $2002; the wages of 
nonrespondents averaged $1836. Earnings 
estimates based on data gathered from 
survey respondents will overestimate the 
actual level of terminee earnings. 
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Table 5.3 displays the results obtained 
when the respondent and nonrespondent 
groups were subdivided by their 
employment status at termination. This 
analysis is of special interest because 
procedures used to correct for nonresponse 
bias in the survey data focus on employment 
status at termination. 

Those employed at termination are more 
likely to respond to the survey and be 
employed during the first full quarter after 
leaving the program than those 
unemployed at termination.^ The higher 
levels of employment for those participants 
who were employed at tennination justifies 
the concern that the different response rates 
of the two groups could inflate estimates of 



post- program employment based on the 
survey data. 

The second observation concerns the 
differences in the post-program 
employment status of respondents and 
nonrespondents within the two subgroups 
defined on the basis of termination status. 
Although the post-program employment 
experiences of respondents and 
nonrespondents are similar for those 
unemployed at termination, there is a 
marked difference in the post-program 
employment status of respondents and 
nonrespondents who were employed at 
termination. This suggests that respondents 
who were employed at termination are not 
a random subset of all terminees who were 



Table 5 J 

Comparison of the Employment Rate for the First Full Quarter After 
Program Termination for Respondents and Nonrespondents by 
Employment Status at Termination*^ 

Title II-A Adult Survey Population 





Employment SUtus at Termination 


Post-Program Survey Status 


Employed 


Unemployed 


Respondents 


78.2 


38.8 


Nonrespondents 


72.7 


40.1 


Response Rate 


70.2 


49.6 


First Quarter Employment Rate 


76.6 


39.5 



^Employment rates calculated from UI wage record data. 
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employed at termination. Since 
respondents from this group had higher 
levels of employment than nonrespondents, 
the estimates of post-program employment 
for all terminees who were employed at 
termination that are solely based on the 
experiences of survey respondents will be 
upwardly biased. 

The fact that respondents who were 
employed at termination are not a random 
subset of all terminees who were employed 
at termination points to the limitation of the 
procedure used to adjust for nonresponse 
bias. The only source of nonresponse bias 
that is considered by the adjustment 
procedure is the bias that occurs due to 
differential response rates across 
subgroups. In attempting to correct for 
these differences, the procedure assumes 
that within each subgroup the post-program 
experiences of respondents and 
nonrespondents are similar. The evidence 
presented above indicates that this 
assumption is correct only for those who 
were unemployed at termination, and it 
does not hold for those who were employed 
at termination. As a result, the adjustment 
may provide some relief from nonresponse 
bias, but it does not provide a full solution 
to the problem. 

The incidence of nonresponse bias 
produced by the different experiences of 
respondents and nonrespondents within 
subgroups is further demonstrated by 
examining the results for males and females 
displayed in Table 54. Although an earlier 
analysis indicated that the response rate 
difference for males and females is 
statistically significant, the overall 
post-program employment levels for the 
two groups were very similar Moreover, 
within each resjx)ndent category, the male 
and female post-program employment 
levels were approximately equal. However, 
for each sex, there is a large difference 



between the post-program employment 
experiences of respondents and 
nonrespondents, indicating that the survey 
estimates of performance for both sexes is 
upwardly biased. Further evidence 
presented in Table 5.5 indicates that, for both 
groups, this bias is produced by the 
differences found among those who were 
employed at termination. 

It is important to note what would happen 
if the adjustment procedure was used in an 
attempt to correct for the different response 
rates of males and females. Given the 
sinularity in the post-program employment 
levels among male and female respondents^ 
the effect of this adjustment would be 
negligible. In other words, the bias 
produced by the different post-program 
experiences of respondents and 
nonrespondents within each sex category 
would remain unchecked. 

There are two major conclusions to be 
drawT\ from these analyses. First, there is 
ample evidence to suggest that the 
post-program survey data is substantially 
affected by the presence of nonresponse 
bias. While this conclusion is based largely 
on the examination of post-program 
employment experiences, it is suspected that 
the same conclusion would hold if the focus 
was on post-program earnings. The second 
conclusion is that the major source of this 
bias, i*e., the different post-program 
employment experiences of respondents 
and nonrespondents who were employed at 
termination, is not addressed through the 
current nonresponse adjustment 
procedures. The implication of these 
findings is that the estimates of 
post-program performance based on the 
information gathered through the 
post-program survey are not a true 
reflection of actual post-program 
experiences of all JTPA terminees. 
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Table 5.4 

Comparison of the Employment Rate for the First Full Quarter After 
Program Termination for Respondents and Nonrespondents by Sex* 

Title II-A Adult Survey Population 





Sex of Respondent 


Post'Program Survey Status 


Males 


Females 


Respondents 


71.2 


70.4 


Nonrespondents 


62.2 


60.0 


Response Rate 


62.0 


67.5 


First Quarter Employment Rate 


67.8 


67.1 



^Employment rates calculated from UI wage record data. 



Issues Concerning the 
Comparison of UI and 
Post-Prograui Survey Data 

The most basic problem with comparing 
the UI data with the survey data is deciding 
on what 'o compare. A comparison of the 
overall UI match rate and the response rate 
to the survey is unjustified since the two 
rates convey different types of information.^ 
Both rates measure success in loca.mg JTPA 
participants, but the match rate depends on 
the pre- and post-program employment of 
the terminee population. If a wage record 
for an individual is not found in a state's UI 
file during a given quarter, it is virtually 
certain that the individual was not engaged 
in Ul-covered employment during that 
quarter in that state."^ In contrast, the fact 
that an individual does not respond to the 



survey provides no clue to their 
post-program employment status. 

This study focused on the differences that 
emerged when UI data and survey data 
were used to measure the post-program 
experiences of JTPA terminees. A technical 
barrier to performing this analysis is that the 
two data sources use different post-program 
intervals. The survey uses a IS-week 
post-program period that begins with the 
week following a participant's termination 
date. Although the survey asks participants 
the total number of weeks they were 
employed during this period, it focuses on 
employment and earnings data for the last 
week of this 13 week period. UI 
wage-record data report wages for a 
complete fiscal quarter defined in terms of 
calendar dates not tied to the participant's 
week or termination. This difference in 
measurement interval makes it ill advised to 
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Table 5.5 

Comparison of the Employment Rate for the First Full Quarter After Program 
Termination for Respondents and Nonrespondents by Sex and Employment Status at 

Termination* 

TiUe Il-A Adult Survey Population 



I. Employed at Termination 

Sex of Respondent 


Post-Program Survey Status 


Males 


Females 


Respondents 


76.8 


79.2 


Nonrespondents 


70.7 


74.7 


Response Rate 
First Quarter Employment Rate 


66.7 
74.8 


73.1 
78.0 


II. Unemployed at Termination 

Sex of Respondent 


Post-Program Survey Status 


Males 


Females 


Respondents 


42.8 


36.5 


Nonrespondents 


44.0 


37.3 


Response Rate 
First Quarter Employment Rate 


52.2 
43.5 


52.2 
36.9 



'Employment rat' j ctJoJated from Ul wage record data 
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compar^ the two sets of data for all 
tenninees. 

One solution is to focus on survey 
respondents whose 13- week post-program 
period corresponds with a fiscal quarter. 
This is accomplished by identifying those 
Title n-A terminees who left the program 
one week prior to the week containing the 
starting date of a fiscal quarter.* 

This solution does not address a second 
problem which emerges because UI 
earnings are reported for the quarter they 
are paid, not the quarter they are earned. As 
a result, UI earnings may appear for 
participants who were not employed during 
a given quarter or may not appear for 
participants who were employed during a 
quarter. For example, if an individual 
became unemployed during the last week of 
a quarter, a portion of the wages earned 
during that quarter may not be paid and 
reported by the employer until the next 
quarter. Similarly, if an individual began 
employment during the last week in a 
quarter, wages earned during that quarter 
may not be paid and reported until the next 
quarter. 

The only post-program measure we could 
replicate in both sets of data was the one that 
identified the terminee as employed at some 
point during the first full post-program 
quarter. For the UI data, we determined 
whether there were nonzero wages for the 
terminee during that quarter. For the survey 
data, we examined the responses to the 
question concerning the number of weeks 
worked during the 13- week post-program 
period and flagged terminees who claimed 
they worked one week or more during that 
period. 

We did not ^re the earnings 
information proviu. _ by the two data 
sources because their wage reporting 
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intervals are different. Although it is 
tempting to use the survey data to estimate 
quarteriy earnings, this procedure makes 
the unfounded assumption that 
respondents' earnings for each week were 
the same as those for the thirteenth week. 
Furthermore, there is the issue of how to 
treat respondents who claimed they worked 
during the post-program period but not in 
the thirteenth week. In these cases, there 
would be no survey data on which to base 
the estimate of quarterly earnings, but these 
earnings would be reflected in the UI data. 
The errors introduced by attempting to 
nuike the UI and survey data comparable on 
earnings would undermine the validity of 
any differences that emerged. 

The project's requirement that survey 
information include the number of weeks 
worke-i in the post-program period posed a 
major problem in using the data from two 
states and a minor problem in a third. SiiKe 
one state did not supply this information at 
all and another did not provide it for 
respondents who indicated they were 
unemployed during the thirteenth week, 
neither state could be included in the 
analysis. Furthermore, a third state's survey 
data for terminees who left the program 
during the third quarter of 1986 was of 
dubious quahty. An unusually high 
percentage of these terminees claimed that 
they were not working during their first 
post-program quarter, but the UI data 
contained a wage record for them. To avoid 
the possibility of contamirwting the results 
of the aiuilysis, we excluded third-quarter 
terminees from this state. Of the 35,415 
remaining terminees selected for 
peirticipation in the post-program survey in 
the nine states, 1,285 participants (3.6%) 
tenninated from JTPA during the required 
interval. That is, their termination week was 
the last week of a fiscal quarter, which makes 
their 13 week post-program period virtually 
coincident with the next fiscal quarter. This 
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group consisted of 863 survey respondents 
and 422 nonrespondents. 

Comparison of UI Wage 
Data to Participant 
Follow-up Data 

The two data sources placed over 
four-fifths (81.7%) of the 863 terminees who 
responded to the 13- week survey into the 
same employment status categories (Table 
5.6). This result reflects a higher level of 
correspondence than Zomitsky et al. (1985) 
found in a similar analysis based on data 
from the state of Maryland (70.3%). 
However, this difference could result from 
the different procedures used to identify the 
terminee subgroups.^ 



Although both .ata sources indicated that 
the majority of terminees were employed 
sometime during the first full quarter after 
termination, the employment levels they 
depicted were different. Survey data 
disclosed that 86% reported being 
employed at least one week during the 
quarter. However, UI files contained wage 
records for only 77.9% of these terminees. 
The reason for this difference is that there 
were many more terminees who claimed 
they worked for whom there was no UI 
wage record (13.2%) than there were 
terminees who claimed they didn't work for 
whom there was a wage record (5.1%). 

Assuming that employers do r.ot report 
wages and therefore pay taxes for 
individuals they do not employ, it is 



Table 5.6 

Comparison of the Employment Status Results for the 
13 Week Program Survey and UI Wage Record Data 



Title II-A Adult Survey Population 





First Quarter UI Status 




Post-Program Survey Status 


Employed 


Unemployed 




Employed 


628 


114 


742 


(7Z8%) 


(13.2%) 


(86.0%) 


Unemployed 


44 


77 


121 




(5.1%) 


( 8.9%) 


(14.0%) 




672 


191 


863 




(77.9%) 


(22.1%) 


(100%) 


Overall Correspondence = 705 (81.7%) 







'Employment rates calculated from UI wage record data. 



ERIC 



JTPA/UI Evaluadon Study 



disturbing that for some respondents who 
claimed that they did not work during the 
quarter but a UI wage record emerged 
anyway. Some of these individuab may 
have deliberately misrepresented their 
employment status, but there is little 
evidence that misrepresentation is 
widespread. In fact, the level of 
misrsporting among welfare recipients, the 
group with the strongest motivation for 
denying employment, was very close to the 
level found for nonrecipients (5.4% and 
4.4%, respectively). 

There are two likely reasons for the 
discrepancy. First, respondents actually 
may have earned the UI wages reported for 
this quarter during the previous quarter and 
subsequently lost their jobs. Second, 
respondents may ha ve experienced memory 
decay, an important source of response 
errors. Both memory decay and the 
reporting of previously earned UI wages are 
possible given the fact that 29 of the 44 
terminees who failed to verify UI reported 
employment were listed as being employed 
at termination from the program. However, 
unless one is willing to argue that all 29 
terminees lost their jobs in less than the one 
week that elapsed between their tennination 
from the program and the beginning of the 
post-program period, the inevitable 
conclusion is that memory problems played 
some role in the responses of this group. 

Memory decay is only an issue with the 
survey question asking terminees to recall 
the number of weeks they worked during 
the post-program period. It is extremely 
unlikely that this problem had any effect on 
the responses to the question concerning 
employment during the thirteenth week. 

There is little reason to suspect memory 
errors among the group of survey 
respondents who claimed they worked in 
the post-program period but for whom a 



wage record could not be found. On the 
other hand, there was a concern with the 
possibility of telescoping effects, i.e., the 
tendency of some survey respondents to 
report that an activity took place in a recall 
period when it actually took place outskle 
the period. However, the vast majority of 
terminees in this group were employed at 
termination from the program (87.7%), and 
a large percentage claimed they worked 
during the thirteenth week (74.6%). These 
two results suggest that memory errors did 
not play a major role in their responses. 

Another explanation is that some of these 
respondents worked in the latter part of the 
quarter liut did not receive their wages imtil 
the next quarter. However, almost 88% of 
these respondents were employed at 
termination, so we suspect that the 
incidence of this problem is fairly low. 

A number of alternative conditions can 
explain why an individual can be employed 
but not included in the state's UI data 
system. First, a person may be commuting 
to an out-of-state job or may have moved to 
another state and obtained employment. In 
either case, the earnings would be recorded 
on the UI system of the state where the 
individual worked. 

Although limitations of the UI database 
prevent an accurate estimate of out-of-state 
employment, it may account for a significant 
percentage of the respondents in this group. 
For these states, the clearinghouse had 
information on the location of the job at 
placement. Of the 33 terminees from three 
states who claimed they were employed on 
the survey but for whom there was no UI 
wage record, 5 (15.2%) were employed 
outside the state at termination. Assuming 
that these terminees were also employed 
out-of-state at the time of the survey, 
out-o^-state employment can account for a 
significant number of respondents. 
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This estimate is likely to understate the 
actual incidence of out-of-state employment 
among the 114 respondents who responded 
as employed in the post-program period but 
for whom no UI wage record was found. 
Over 28% of these respondents were from a 
state with a major out-of-state metropolitan 
area on its border. For this state^ out-of-state 
placement is more likely to be an issue than 
in the three states we examined. As a result, 
15% is a conservative estimate of the level of 
out-of-state employment represented in this 
database. 

Second, individuals may have jobs not 
covered by UI. Earnings may be exempt 
from state and federal reporting 
requirements because the terminee is 
self-employed or is working for an employer 
who is not required to report wages. 
However, UI reporting requirements are 
such that the vast majority of workers whose 
earnings are exempt are likely to be 
self-employed (Hanna, 1989). Another 
reason that workers may not appear in the 
UI files is that they are paid under the table. 
There would be no official record of 
earnings on the UI system, but the 
individual may be willing to report earnings 
during a confidential survey. 

A rigorous analysis of the relative roles of 
these two types of noncovered employment 
would require three procedures: (1) to 
remove those employed out-of-state from 
the database, (2) to identify self-employed 
individuals, and (3) to distinguish 
employers who are subject to UI rejwrting 
from those who are not. The current 
database does not contain the detailed 
information needed to support these 
procedures. 

Can the limitations of UI wage records be 
addressed so that they can confidently be 
used for performance assessment within a 
state? First, if the purpose of using UI data 



is to draw state-level inferences, out-of-state 
employment is a minor issue in many states. 
In the three states where it was i)0ssible to 
identify the location of the job at 
termination, less that 5% of all employed 
terminees had jobs in other states. Thus, 
out-of-state employment is small and is 
unlikely to affect state-level estimates of 
post-program employment. 

On the other hand, if the purpose of using 
UI data is to draw SDA-level inferences 
(which are necessary to produce SDA 
performance standards), it may be 
important either to gather supplemental 
information on those employed out-of-state 
or to introduce adjustment procedures to 
account for their occurrence. The SDA-level 
results in the three states with out-of-state 
placement information indicate that those 
employed out of state were not equally 
distributed across the SDAs. If participants 
employed out of state have markedly 
different employment experiences from 
those employed within the state, it is 
important to address the issue of out-of-state 
employment. 

The problems associated with out-of-state 
employment can be corrected by developing 
data sharing agreements among states. The 
evidence from the three states indicates that 
over 99% of out-of-state employment occurs 
in neighboring states and that out-of-state 
employment in noncontiguous states is 
randomly distributed across all SDAs. 
Thus, efforts can focus on the collection of 
data from contiguous states for the few 
participanis employed in those states at 
ternunation. The fact that a nimiber of states 
participating in this project have provided 
each other with data supports the feasibility 
of this approach. Furthermore, the 
expansion of the Internet pilot program, 
which allows one state to crossmatch UI 
claims against the UI records of other states 
to detect possible cases of fraud, suggests 
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that data exchanges are becoming a routine 
activity among many states. 

Other approaches to the problem of 
out-of-state employment are available. For 
example, SDAs with a high incidence of 
out-or -state employment could conduct 
surveys to provide the infonnation needed 
to adjust the Ul-based estimates of 
post-program performance. Since these 
surveys would target only those terminees 
employed out of state, they should not be 
costly. 

Neither data exchanges nor adjustment 
procedures, however, may be required to 
draw accurate and reliable post-program 
performance estimates for SDAs. When we 
compared the foUow-up entered 
employment rates for participants 
employed out-of-state at termination and 
those employed within the state, we found 
that the rates for the two groups differed by 
less than 0.5 percentage points. This 
suggests that gathering information on 
terminees employed out of state is probably 
not ne essaiy because their post-program 
experiences are likely to be similar to those 
employed in-state. Therefore, SDA-level 
estimates based on participants employed 
in-state at termination might provide an 
accurate reflection of the performance of all 
terminees for an SDA. This prehminary 
conclusion, however, requires further 
investigation. 

Estimates of performance based on UI data 
can be adjusted to account for terminees 
who are either sei^-employ ed or working for 
employers who are not required to report 
wages to the UI system. The proposed 
solution requires that states identify these 
two types of employment at termination. Ii 
information collected at termination 
indicates that the incidence of these types of 
employment is low or that they are 



randomly distributed among SDAs, such 
adjustments would not be necessary. 

The coverage issue surrounding those 
terminees who were paid under the table, 
i.e./ those who worked in a cash economy, 
deserves special attention. This problem 
has solutions, but it is not dear that they are 
needed. Virtually all jobs considered 
appropriate for JTPA participants are 
covered by UI, and employer compliance 
with UI laws is high. Thus under-the-table 
employment should be rare and randomly 
distributed across SDAs. 

The possibility that under-the-table 
employment is not random leads to an 
interesting policy question. Should 
adjustments be introduced to provide credit 
to those SDAs whose terminees are more 
likely to be found in this type of 
employment? On the one hand, it can be 
argued that if an SDA is in an area with a 
higher level of noncompliance among 
employers, it should not be held accountable 
for this factor. On the other hand, such 
employers deny their employees the 
benefits to which they are entitled under 
state law. Adjustments might legitimize 
under-the-table employment as a desirable 
JTPA outcome. 

The issue of under-the-table employment 
points to an important advantage of using 
UI wage records to assess post-program 
perfomrumce. It guarantees that the jobs 
obtained by JTPA terminees will provide a 
minimum level of benefits. The 
post-program survey reports any job held in 
the thirteenth week regardless of its 
quality-even a once-a- week babysitting job. 

The movement to a Ul-based assessment 
system is likely to deaease the incidence of 
employment not covered by UI in the Jl I'A 
population. Realizing that covered 
employment is the key to unlocking 
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incentive funds, JTPA service providers will 
focus their attention on providing their 
clients with services that can lead to such 
employment. Furthermore, they will 
monitor the employers of their participants 
more closely to ensure that they are 
providing the benefits required by law. 

Cost Comparisons 

The start-up costs for designing and 
implementing a system for gaining access to 
UI wage-record data will vary depending on 
the size of the state program. At the high 
end, one state in ihe project estimated that it 
cost $50,000 to $60,000 to develop its 
Ul-based employment tracking system. 
However, this system is relatively 
sophisticated and can supply UI data to all 
state agencies, not just the one administering 
the JTPA program. Three other states 
providing information on their start-up 
UI/JTPA matching costs estimated their 
costs to be approximately $20,000, with a 
low of $18,257. The experience of these three 
states suggests that other states should 
expect a minimum expenditure level of 
around $20,000. 

Estimated maintenance costs were much 
lower. No state estimated that it would cost 
more that $10,000 per year to access UI data 
for new terminees and to update existing 
files with additional quarters of 
post-program data. In fact, one state 
estimated that it would cost only $2,700 per 
year. 

The estimated costs of collecting the 
post-program survey data are much higher. 
In some stales, they are quite high. The state 
reporting the lowest cost for the survey 
estimated $75,000 as necessary to collect 
follow-up data in PY89. Another state 
estimated that its survey costs exceed 
$230,000 per year. Based on its estimate of 



the cost of collecting UI data, this last state 
would save over $200,000 if there was a shift 
to a Ul-based post-program system. 

The more a state spends to gather 
post-program survey data, the more it 
stands to gain in the movement to a 
Ul-based assessment system. This is a 
simple function of the fact that maintenance 
costi are expected to be below $10,000 for all 
states. Even in states where the cost of 
post-program data collection is relatively 
low, a substantial reduction in the total cost 
of collecting data should be reahzed. 

To provide a better understanding of the 
nationwide impact of these cost 
differentials, the clearinghouse compared 
the total costs of the two options based on 
start-up and maintenance costs of the six 
states providing information. The estin\ate 
for start-up costs for accessing UI wage 
records was based on the assumption that 
they would be a linear function of the JTPA 
Title II-A allocation for PY88. We also 
assumed that no state would have UI 
start-up costs of under $20,000 and set lower 
estimates to this amount. Maintenance costs 
were simply set at $10,000 for each state. 
Both sets of state estimates were then 
summed to produce the national estimates 
of $1,394381 for start-up and $500,000 for 
annual maintenance of the system. 

The clearinghouse roughly estimated the 
national costs of the post-program survey as 
follows. First, we estimated the minimum 
sample size for the Adult Title II-A survey 
for each state's SDAs using PY87 data on 
program terminations. This produced a 
nationwide estimate of approximately 
161,000 sampled participants. Assuming a 
response rate of 70%, we estimated the 
number of respondents to be 112,700. We 
then multiplied this figure by the average 
cost per completion among the participating 
states, i.e., $25.90.^ This produced a total 
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estimated cost of $2,918,930 for collecting 
post-program data for the adult sample for 
one year. 

The three sets of results (Figure 5.1) 
dramatically illustrate the cost effectiveness 
of the UI approach. The cost of 
implementing a national Ul-based data 
collection system is estimated to be less than 
one-half the current cost of collecting survey 



data, and the ongoing costs of maintaining 
the system are estimated to be less than 
one-fifth. Actual costs of conducting the 
follow-up survey are underestimated in this 
analysis because (1) the supplemental 
welfare sample is not cor\sidered, (2) the cost 
of Title in data collection is excluded, and (3) 
some states do not sample but survey their 
entire JTPA population. 



Figure 5.1 

Comparison of Ul and Post-Program 
Survey Data Collection Costs 
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Another factor to consider in comparing 
costs is the quantity of data purchased with 
these funds. The post-program survey 
provides one week's worth of employment 
and earnings information on a sample of one 
year's terminees. UI data provides multiple 
quarters of pre- and post-program data for 
virtually all of those terminated from the 
program in a given year plus additional 
post-program information for terminees 
from previous years. If the two alternatives 
are compared solely on the basis of the 



quantity of information obtained, the UI 
approach has a decided advantage. 

The impact on costs of new post-program 
data requirements for the Title III program 
must also be included. Previously, states 
were required to conduct a statewide survey 
of Title ni participants. The new federal 
reporting requirements call for 
representative surveys conducted by or for 
each Title III substate grantee. Since the 
n\inimum sample-size criterion used for the 
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Title II-A surveys will apply to these surveys 
as well, the total number of JTPA 
participants needing to be contacted will be 
substantially increased in each state v^th 
more than one substate grantee. 

The costs of survey research will keep 
escalating because it is a labor-intensive 
activity. The major expense is the wages 
paid to the interviewers. As wages rise, so 
does the cost of conducting the survey. New 
minimum wage legislation will almost 
certainly affect the costs of the post-program 
survey. 

The collection of UI data depends on 
technology. Continuing advances in data 
processing technology are likely to keep 
costs at a reasonable level. In fact, it may 
even beccme less expensive to access this 
information in the future. The costs of 
collecting and storing this information are 
lower now than they were ten years ago. 
This suggests that the cost differential 
between collecting survey data and 
accessing UI data can be expected to increase 
over time. 

Siimmary 



Three basic criteria can be used to judge the 
relative quality of the post-program survey 
data and UI data. The first criterion is the 
extent to which each data source can 
produce unbiased estimates of SDA 
performance. The lesi biased an estimate, 
the more likely it is to reflect the actual 
performance of an SDA. The potential for 
bias in Ul-based performance measures is 
minimal given the high percentage of 
employment covered under UI regulations. 
Furthermore, noncoverage problems are 
either random or they can be handled with 
adjustment procedures. 



In contrast, the estimates of post-program 
performance produced by the survey data 
are substantially affected by nonresponse 
bias. Furthermore, the type of bias found is 
left unchecked by current adjustment 
procedures and tends to inflate estimates of 
performance. To a lesser extent, 
post-program survey data were affected by 
the memory recall problen\s of respondents. 
This was not a problem with the measure of 
employment during the thirteenth week, but 
it affected the measure focuf Ing on the 
number of weeks worked in the 
post-program period. 

The second criterion refers to the precision 
or reliability of the estimates that are 
produced. Although an estimate may be 
unbiased, this does not mean it is reliable. 
Reliability is priniarily a function of the 
amount of data available to produce the 
estimate. The more data, the higher the level 
of confldence in the estimate- 
Since Ul-based performance estimates are 
calculated on a database that can potentially 
represent all terminees, high precision can 
be expected from these estimates. Thus, the 
reliability of Ul-based post-program 
performance measures is essentially not an 
issue. Furthermore, the reliability of the 
performance estimates for each SDA should 
be approximately the same. As a result, a 
high level of confidence can be placed in the 
decisions made on the basis of UI 
information. 

The precision of the performance estimates 
based on survey data is lower for two 
reasons. First, most states and SDAs survey 
a sample of terminees rather than their entire 
terminee population. Because fewer cases 
are used to produce the estimates, the 
precision of the estimates is lower. In 
addition, the number of nonresponses to the 
survey also serves to lower the number of 
cases and, therefore, the reliability of the 
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estimates. A particular concern with the 
post-program survey is a response rate too 
low to achieve the ininimum sample size. 

The minimum sample size requirements 
assume a response rate of 70%. SDAs tend 
to achieve different response rates for a 
variety of reasons^ and in PY87 this resulted 
in a range of SDA response rates that varied 
by over 30 percentage points (Dickinson and 
West, 1989). There is likely to be a great deal 
of variation in their precision of the SDA 
performance estimates if the SDAs do not 
adjust their sample size to account for the 
particular response rate they achieve. 

The final criterion addresses the quality of 
the data. Quality depends in part on 
whether the same procedures and level of 
effort are used to collect the information in 
each SDA. If these are the same, then the 
quality of the data is likely to be consistent. 
If not, then there are likely to be difficulties 
in drawing comparisons among SDAs. 

There is no question that the quality of UI 
data will be highly consistent within a state. 
However, the coverage requirements of 
state UI laws differ slightly, and there are 
differences as well in the way states collect 
and process information. Some adjustments 
may be necessary before the data can be 
pooled across states for analysis. 

Whether post-program survey data are 
consistent within a state largely depends on 
whether the collection is centralized or 
decentralized. If collection is centralized 
and a reputable survey research firm 
conducts the survey, the data are probably 
consistent. However, if each SDA is 
responsible for collecting its own data, there 
is always the risk that the quality of the data 
will vary from SDA to SDA. This 
probability is increased when the state fails 
to monitor the data collection activities of its 



SDAs through some form of validation 
procedure. 

Consistency is an issue at the national level 
as well as at the state level. The federally 
mandated post-program survey can 
actually be viewed as over fifty separate 
surveys. Despite all precautions, the quality 
of the data probably varies across the states. 
The extent of this variation and its possible 
impact on the development of performance 
standards models remain to be examined. 

On each of the three criteria for judging 
data quality (lack of bias, precision, and 
consistency), UI wage record data are found 
to be superior to post-program survey data. 
Concerns with regard to the use of UI data 
seem minor compared to the problems 
identified with the post-program survey 
data. Even if cost were not an issue in 
selecting a source of data, concern over the 
quality of the information argues strongly in 
favor of the UI alternative. 

The relative costs of these two options for 
collecting data only serve to strengthen the 
argument for Ul-based performance 
assessment. The ongoing costs of accessing 
UI wage-record data can be conservatively 
estimated as less than one-fifth the cost of 
collecting post-program data through 
participant contact. For a lower cost, the 
JTPA system can expect to receive more 
information on its participants than the 
survey provides. Such data will include 
valuable pre-program infonnation plus the 
post-program data necessary to develop 
long-term performance indicators. 

In this era of deficit reduction, it is evident 
that JTPA must seek cost-effective ways to 
fulfill its mission. One way to reduce costs 
and Improve the capacity to monitor the 
system is to move toward a Ul-based 
assessment system. 
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Chapter Endnotes 

1. This evidence suggests that DOL was correct in assuming that these two groups would 
have different response rates to the survey. 

2. The overall I Jl match rate is defined as the percentage of participants found to be employed 
in any of the pre- or post- program quarters. Since one of the most likely sources of a 
non-match is that the terminee was unemployed during both the pre-and-post-programs 
periods, the 88.6% match rate for Title II-A adult terminees and the 95.3% match rate for Title 
ni terminees probably understates the overall success of this project in tracking JTPA partic- 
ipants through state UI files. 

3. Although a nonmatch does not eliminate the possibility that the individual was employed 
in a different state or in a job not covered by VI, it substantially limits the range of alternatives. 

4. This procedure allows for some minor problems to be introduced because the 13-week 
follow-up period may not perfectly coincide with the ensuing fiscal quarter. 

5. The comparison group for the analysis of Maryland's data was based on two quarters of 
information and required that positive earnings be recorded for both quarters. 

6. The average cost per interview among the participating states is very close to the average 
cost found in a survey of 42 states conducted by the National Governors' Associatic n in 1987. 
This survey found that the average cost per completion in PY86 was $33.20. For those states 
that began conducting follow-up surveys before October 1986, the cost was $25.10. 
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Pre- and Post-Program Employment and 
Earnings: Patterns of PY86 Title II-A 
Adult Terminees 



One of the major advantages of UI 
wage-record data is its flexibility. Chapter 6 
illustrates this flexibility by examining the 
pre- and post-program employment and 
earnings patterns of PY86 Title D-A adult 
terminees from two distinctly different 
perspectives. The first section focxoses on 
selected demographic subgroups, and the 
second focuses on experiences of 
participants grouped according to program 
interventions. 

Prior to presenting these findings, it is 
necessary to offer several words of caution 
regarding the interpretation of the results of 
these analyses. As noted earlier. The 
National Commission for Employment 
Policy convened a panel of technical experts 
to review and comment on the goals and 
objectives of this project. The panel voiced 
concerns that are particularly relevant to 
analyses discussed in this chapter. Findings 
from the simple descriptive analyses 
presented are often taken out of context. In 
particular, the panel was concerned that 
differences between pre- and post-program 
employment and earnings would be 
misinterpreted as evidence of the net impact 
of the JTPA program. There is a natural 
temptation to attribute gross changes in 
employment rates and average earnings to 
program participation. However, such 
conclusions are premature. Many other 
factors also contribute to these differences. 
For example, changes in local labor market 



conditions are likely to have a significant 
impact on observed differences in pre- and 
post-program employment rates and 
average earnings. Unless these factors are 
carefully controlled, their effects will 
mistakenly be attributed to program 
participation. 

The fact that the following analyses cannot 
be used to draw inferences concerning the 
net impact of the JTPA program cannot be 
overemphasized. In order to estimate net 
impact, it is necessary to identify the 
incremental gains in employment and 
earnings of JTPA participants that occur 
over and above what would have happened 
if they had not participated in the program. 
In other words, the net impact of the 
program can only be assessed after the 
effects of other factors have been removed 
from the gross differences observed 
between pre- and post-program periods. 

The only guaranteed way to remove 
nonprogram effects is to use a research 
design that directly relates the experiences 
of program participants to a control group 
of nonparticipants. For example, net impact 
studies employing classical experimental 
designs randomly select program 
participants from the pool of people eligible 
for the program and place those not selected 
in a control group. Another approach is to 
use a quasi-experimental design. These 
studies typically use administrative data 
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sources to assemble a comparison group (in 
lieu of a pure conUx>I group) and rely on 
statistical methods to adr for the inherent 
differences between tho v .iparison group 
and program participants. 

Regardless of which net impact design is 
employed/ the objective is to use the 
experiences of non-participants to 
approximate what the post-program labor 
market experiences of participants would 
have been if they had not participated in the 
program. Since the database developed for 
this project consists solely of program 
participants, it caimot address questior\s 
about the net impact of the ]TPA program. 
However, this capacity can be developed. 
Some of the states participating in this 
project have undertaken the task of 
coi\structing comparison groups and are 
currently examiiiing the net impact of JTPA 
in their states. 

A related coiKem of the panel focused on 
the possibility that the findings produced 
from the subgroup analyses would lead to 
judgments that JTPA serves one subgroup 
better than another. The findings presented 
in this chapter are ii^ffident to support this 
type of conclusion because evidence to 
support it can be obtained oiUy through a net 
impact design. 

There are additional dangers in attempting 
to compare outcomes from program 
interventions. Participants are not 
raiKionUy assigned to program activities. A 
myriad of factors influence assignments, 
including the preferences and needs of the 
client and the range of services offered by 
the SDA. Consequently, the characteristics 
of participants vary by program activity, 
and these differences complicate the 
comparison of outcomes. For example, it is 
impossible to determine from a simple 
comparison of two interventions whether 
the differences in their outcomes is a 



function of the inherent quality of the 
intervention of the differences in the types 
of participants receiving the interventions, 
e.g., welfare recipients versus nonredpients. 

The usefulness of these comparisons is not 
that they enable one to draw conclusions but 
to raise questions and issues. For example, 
these comparisoiis can focus attention on 
specific aspects of subgroup equity. Thus, 
although defitUtive conclusior\s cannot be 
drawn, the following analyses can 
contribute to ongoing debate by refining the 
issues and acting as a springboard for 
further research. 

Subgroup Analysis 



Figure 6.1 presents the pre- and 
post-program employment and earnings 
patterns for all PY86 Title II-A temrinees. 
Although the average pre-program eanUngs 
declined sharply during the four quarters 
prior to enrollment, employment rates 
based on UI data showed relatively little 
change. These results, in part, reflect the fact 
that eligibility for the program is determined 
on the basis of income rather than 
employment. In addition, the relatively 
consistent employment patterris coupled 
with the dramatic decline in earnings 
suggests that a large number of terminees 
may have moved from full-time to part-time 
employment prior to entering the JTPA 
program. 

The post-program employment and 
earnings patterns were also different. The 
average UI earnings of those employed 
increased over the post-program period, but 
the quarterly employment rates declined. 
These findings warrant a more detailed 
analysis of the relationship between 
employment and earnings. Subsequent 
phases of this project will cover PY87 and 
PY88, and they will extend the follow-up 
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period to eight quarters after termination for 
PY86 and PY87. These expanded data sets 
will support a closer analysis of the labor 
n\arket experiences of JTPA participants. 



including the surprisingly large dips in 
employment rates between the third and 
fourth post-program quarters, which are 
illustrated in Figure 6.1. 



Figure 6.1 

Employment and Earnings Trends 
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The pre- and post-program employment 
trends of males and females were 
remarkably similar (Figure 6.2). Although 
females had lower employment rates than 
nrtales for each of the quarters examined, 
these differences were not large, especially 
during the post-program period. 

In contrast, the average earnings of males 
and females indicate large differences 
between the two groups in each 
pre-program and post-program quarter 
(Figure 6.3). Males, as expected, had higher 
average earnings. Although the difference 
between male and female earnings declined 
in the pre-program period, it increased over 



the course of the post-program period. The 
rate of increase was not large, but it is fairly 
constant. 

The pre- and post-program trends by race 
and ethnic :jtatus revealed some surprising 
results. Three groups, Hiiipanics, American 
Indians and Alaskan Natives, and Asian and 
Pacific Islanders, had higher levels of 
employment in the quarters immediately 
prior to program enrollment than they did 
in earlier quarters (Figure 6.4). 

The earnings for all racial and ethnic 
groups declined in the pre-program period 
(Figure 6.5). The rate of decline was roughly 
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Figure 6.2 

Employment Trends by Sex 
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Figure 6.3 

Pre/Post Average Earnings Trends by Sex 
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Figure 6.4 

Pre/Post Employment Trends by Race 
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Figure 6.5 

Pre/Post Average Earnings Trends by Race 
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the same for all groups. The average 
earnings for all groups increased over the 
course of the post-program period. In 
addition, the relative rankings of the groups 
remained the same throughout this period. 

The employment rate and earnings of 
Hispanics were higher than expected in both 
the pre- and post-program periods. 
However, these findings are consistent with 
the findings of a recent study of the National 
Commission for Employment Policy (1990) 
focuising on the participation of Hispanics in 
JTPA programs. The study cited evidence of 
a strong work ethic in the Hispanic 
community, leading its members to accept 
low-wage jobs in preference to being 
without work. If their work ethic also 



prompts Hispanics to work more hours in 
the jobs they do obtain, it can account for the 
employment and earnings patterns found in 
the current study. 

Pre- and post-program employment trends 
for the three groups defined on the basis of 
their educational status at enrollment were 
similar (Figure 6.6). The employment levels 
of high school gradtiates and post-high 
school attendees were roughly the same, but 
dropouts had substantially lower levels of 
employment in each of the pre-program and 
post-program quarters. Furthermore, the 
differences between dropouts and the other 
two groups were greater in the 
post-program period than in the 
pre-program period. 



Figure 6.6 

Pre/Post Employment Trencis 
by Educational Status 
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Pre- and post-program average earnings 
for these three groups were also similar 
(Figure 6.7). Earnings levels for high school 
graduates and post-high school attendees 



were about the same in the pre-program 
period, but some differences emerged in the 
post-program period. Participants with 
some post-high school education had 
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substantially higher earnings in each much lower earnings in each of the 
post-program quarter than those without post-program quarters, 
post-secondary education. Dropouts had 
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Pre/Post Average Earning Trends 
by Educational Status 
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The final subgroup comparison focused on 
the welfare status of participants at 
enrollment. Figure 6.8 illustrates that there 
was a large difference in the pre-program 
employment levels of welfare recipients and 
nonrecipients. There were also differences 
in the post-program employment levels/ but 
they were much smaller. There were larger 
differences in the average earnings of 
welfare recipients and nonrecipients in the 
pre-program period than in the 
post-program period (Figure 6.9). 

Recipients of Aid to Families with 
Dependent Children (AFEXZ) had higher 
levels of employment in each of the 
pre-program and post-program quarters 
than recipients of other types of public 
assistance. On the other hand, AFDC 



recipients had lower earnings (Figure 6.9). 
These results are probably a function of the 
sex composition of these two groups. AFDC 
recipients are typically female; those 
receiving other forms of public assistance 
are largely male. 

Activity Analysis 

Access to activity data provided the 
clearinghouse with the opportunity to 
define participant groups on the basis of the 
types of program interventions they 
received. The clearinghouse translated each 
state's codes of these activities into a 
common coding scheme. Since states 
provided starting and ending dates for each 
intervention, it was possible to identify each 
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Figure 6.8 

Pre/Post Employment Trends by Welfare Status 
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Pre/Post Average Earnings Trends 
by Welfare Status 
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participant's sequence of activities. The 
activity sequence indicator plays a 
prominent role in the analyses presented in 
this section. 

The indicator was designed to reflect 
movements across different categories of 
interventions. As a result, it is insensitive to 
movements vyrithin a particular category 
when those interventions are received 
consecutively. For example, suppose a 
participant was consecutively enrolled in 
two different forms of occupational 
classroom training and subsequently 
received job search assistance. The activity 
sequence indicator will ignore the fact that 
the participant transferred from one 
classroom training intervention to another 
and will reflect only the movement from 
classroom training to job search assistance. 
However, if job search assistance had begun 
before enrollment in the second classroom 
activity, the indicator would reflect 
movement across all three activities, i.e., 
from occupational classroom training to job 
search assistance and back to occupational 
classroom training. 

No technical reason forced this design of 
the activity sequence indicator. It could have 
been constructed to reflect all interventions. 
However, for this report, we decided to limit 
the analysis to movements across different 
categories of interventions and to ignore 
movements within these categories. 

Given that the indicator is based on the 
chronological order of the interventions, it is 
important to describe how we treated 
concurrent and overlapping activities. 
JTPA participants can receive different types 
of interventions simultaneously. We treated 
overlapping activities (where a new activity 
began before another ended) as if they 
occurred at two different times. Concurrent 
activities (where the starting dates for the 
two activities were the same) were more 



difficult because it was unclear which 
should appear first in the sequence. The 
procedure we used was to define the 
intervention with the earliest exit date as the 
first activity. 

To focus attention on JTPA's major types of 
program interventions, we collapsed the 
ninecategories of interventions employed in 
the original coding scheme to five 
categories. First, we eliminated the holding 
category from consideration because it is not 
a program intervention but a waiting state 
that the participant enters prior to receiving 
subsequent interventions. Second, we 
collapsed the classes of activities previously 
identified as (1) work experience, (2) 
employment and training services, and (3) 
youth activities into the "other" category. 
The five major classes of interventions that 
remained were: 

• Occupational Classroom Training. 

• Remedial/Basic Educational 
Classroom Training. 

• On-the-job Training. 

• Job Search Assistance. 

• Other. 

All the analyses presented in this seciion 
focus on the adult Title II- A population. The 
section examines the experiences of a total of 
100,849 participants; 3,259 were excluded 
because of missing data. 

DISTRIBUTION OF PARTICIPANTS 
ACROSS PROGRAM 
ACTIVITIES 

Table 6.1 shows the percentage of adult 
Title II-A participants who received one or 
two types of program interventions by the 
sequence of those interventions. Sixty-five 
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percent (the sum of the diagonal elements) 
of all adults served by the eleven states 
received only one type of program 
intervention during their stay in the 
program, and 24.6% (the sum of the 
off-diagonal elements) received two types of 
interventions. The remaining 10.8% 
received three or more different 
interventions. 



PY86 participants received on-the-job 
training most often. Nearly onethird 
received this intervention either alone 
(23.7%) or in combination with another type 
of intervention (9.5%). 

Occupational classroom training was the 
second most popular form of intervention, 
and job search assistance was third. By a 
large margin, the least prevalent 



Table 6.1 

Percent of Participants Receiving Interventions by the Sequence of Those Interventions 

PY86 Adult Title II-A Texminees 
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intervention was educational classroom 
training. Only 6.2% of all participants 
received basic educational training during 
their stay in the program. 

The majority of participants who received 
interventions defined as "other" did so in 
conjunction with another activity. This 
category of interventions includes services 
such as vocational counseling and labor 
market orientation that are designed to 
assist participants in other training funded 
under JTPA. It is perhai>s surprising that 
these services are reported for so few 
participants. However, we suspect that they 
are underreported on state MISs. 

If individual program interventioi\s are 
underreported/ then, implicitly, the 
percentage of participants receiving 
multiple interventions is underestimated as 
well. However, the percentage of 
participants in this study reported as 
receiving two or more interventions is 
34.4%, much higher than previously 
published national estimates. An additional 
10.3% received "Other" interventions. Based 
on a national sample of adults who left the 
JTPA program in the first three months of 
PY86, EXDL estimated that 16.4% received 
either multiple or "other" interventions.^ 

TERMINATION OUTCOMES OF 
PROGRAM ACTIVITIES 

The analyses presented in this section do 
not adjust the outcomes from the various 
program activities for differences in client 
characteristics. Thus, if one intervention is 
found to produce a lower outcome than 
another, this does not imply that the first 
intervention was less successful than the 
second. The first may have been used for a 
harder-to-serve group of participants. 

Despite this limitation, an analysis of 
termination outcomes for the various 



sequences of interventions is a useful first 
step towaid understanding the dynamics 
underlying the provision of JTPA services. 
Such an exploration can help define issues 
in need of more serious examination. 

The popiilarity of on-the-job training is 
understandable given that participants 
placed in this activity have a higher 
probability of being employed at 
termination than participants in other 
activities (Table 6.2)} In fact, the five 
highest entered employment rates 
displayed were produced by sequences 
containing on-the-job training as a 
component. Furthermore, when on-the-job 
training is offered as the last intervention in 
a sequence of activities, the entered 
employment rates indicate that it is even 
more effective than when it is offered as the 
only activity. 

Although job search assistance produces a 
relatively high entered employment rate 
when it is used as the sole intervention, there 
are mixed results when it appears in 
conjuiKtion with other interventions. These 
results suggest that job search assistance is 
sometimes used as the intervention of last 
resort, i.e., when the initial intervention does 
not result in the client entering employment. 
This is best illustrated by the sequence in 
which job search assistance follows 
on-the-job training. The low entered 
employment rate that results from this 
sequence indicates that providing job search 
assistance to participants who fail to enter 
unsubsidized employment following 
on-the-job training is seldom effective. 

When offered as the only intervention, 
classroom training activities appear to 
produce lower entered employment rates 
than either on-the-job training or job search 
assistance. However, these results are 
difficult to interpret because the entered 
employment rates displayed in Table 6.2 are 
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not adjusted for the characteristics of the 
participants. In other words, participants 
placed in classroom training may face more 
serious barriers to employment than those 
who are initially placed in on-the-job 
training or job search assistance. Hierefore, 
the relative success of these activities cannot 
be judged through a simple comparison of 
entered employment rates. 

When the average wage at placement is 
examined, a different pattern emerges. 
Table 6.3 indicates that the highest average 
wages ($5.01 ) are found for participants who 



received occupational classroom training as 
their only Intervention. Next in order was 
on-thejob training followed by job search 
assistance and, finally, remedial and basic 
educatioiuil training. If the low entered 
employment rate for occupational 
classroom training can be attributed to 
attrition, these results imply that 
participants who successfully complete this 
form of training are likely to receive higher 
wages in the jobs they obtain after 
termination from the program than 
participants in other interventions. 



Table 6.2 

Entered Employment Rate by the Sequence of Interventions 
PY86 Adult Tide II-A Terminees 







Second Intervention 




First 
Intervention 


Occupational 
CRT 


Remedial/ 
Basic CRT 


On-thejob 
Training 


Job Search 
Assistance 


Other 
Interventions 


Occupational 
CRT 


66.8 




81.0 


62.5 


58.2 


Remedial/ 
Basic CRT 


65.9 


58.2 






* 


On-the-job 
Training 




» 


79.4 


49.3 


81.8 


Job Search 
Assistance 






89.5 


79.1 




Other 

Interventions 


76.7 




89.1 


73.1 


70.9 



Note: 

* = less than .5% received that mix of interventions. 
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Table 63 

Average Wage at Placement by the Sequence of Interventions 
PY86 Adult Title II-A Tenninees 







Second Intervention 




Pi rat 

Intervention 


CRT 


Remedial/ 
Basic CRT 


On-the Job 
Training 


Job Search 
Assistance 


Other 
Interventions 


Occupational 
CRT 


5.01 




4.59 


5.21 


4.7b 


Remedial/ 
Basic CRT 


4.64 


4.41 








On-the-job 
Training 


» 


* 


4.88 


4.96 


5.04 


Job Search 
Assistance 




* 


4.61 


4.77 




Other 

Interventions 


4.84 




5.01 


4.85 


4.80 



Note. 

* 3 les than .5% received that mix of interventions. 



Table 6.3 also illustrates that offering job 
search assistance after other traiiung can 
help participants find jobs that provide 
higher wages. This suggestion is supported 
by the finding that the wages for 
participants who received job search 
assistance as their second intervention in a 
two-intervention sequence are uniformly 
higher than participants who received only 
the first intervention. ITiis implies that job 
search assistance is not always used as a last 
resort but also as a means of helping 
participants take full advantage of the skills 
they developed in earlier training. 



PRE-PROGRAM LABOR FORCE 
EXPERIENCES 

The suggestion that participants are 
assigned to activities on the basis of an 
assessment of need finds support when 
pre-program earnings are used as an 
indicator of this need. Although the 
differences in pre-program earnings among 
the participant groups are not large, there is 
evidence that participants with lower 
pre-program earnings tended to receive 
more interwive JTPA services. Figure 6.10 
illustrates that participants who received 
classroom training as their only intervention 
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had lower pre-program earnings (averaged 
across the four quarters) than participants 
in other interventions. On the other hand, 



the participant group receiving only job 
search assistance had one of the highest 
levels of pre-program earnings reported. 



Figure 6.10 

Mean Level of Pre-Program Average Earnings 

By Program Activity 
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The pre-program earnings of participants 
identified as receiving multiple 
interventions reflect the range of different 
service strategies used with this group. Not 
all combinations of interventions are equally 
intense. For example, vocational counseling 
and job search assistance is a much less 
intense activity than remedial education and 
occupational classroom traiiUng. It is not 
surprising to find that the pre-program 
earnings of the group that receives this 
sequence occupies a middle position in the 
rank order. Thds serves as a reminder that 
the number of interventions received should 
not be confused with the intensity of service. 

The quarter-to-quarter changes in the 
pre-program earnings patterns were similar 
across the different groups (Figure 6.11). 



For all participants, pre-program earnings 
declined from the fourth to the first 
preprogram quarter. The earnings levels of 
the groups converged in the quarters 
immediately preceding enrollment, 
probably because eligibility is based on 
earnings. 

Post-profcram Labor Force Experiences . 
The variation between groups in 
post-program earnings is much greater than 
the variation in the pre-program period 
(Figure 6.12). Comparing the range of 
earnings among the groups for the 
pre-program and post-program periods 
further illustrates this point. The difference 
between the participant groups with the 
highest and lowest pre-program quarterly 
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Pre-Program Quarterly Average Earnings 
by Program Activity 
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Figure 6.12 

Mean Level of Pre/Post Average Earnings 
by Program Activity 
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earnings is $173; the corresponding 
post-program difference is $468. 

In comparing post-program findings, the 
most striking result is the relatively high 
earnings levels of participants who received 
on-the-job training as their only 
intervention. These participants averaged 
over $100 more per quarter than any other 
group. This result could not have been 
prei ."^^d based on the pre-program 
earit i .i- ^f this group. Although the 
pre-pro- m earnings of on-the-job training 
participants were above the average for the 
six groups, two groups had higher 
pre-program earnings than participants in 
on-the-job training. 

At the other extreme are the relatively low 
post-program earnings reported for 
participants receiving only remedial and 
basic educational training. However, this 
finding is understandable given the 
relatively low pre-program earnings of this 
group. 

The post-program performance of 
participants who received only job search 
assistance or "other" interventions was 
unexpected. Given their pre-program 
earnings levels, we anticipated that these 
two groups would have placed high in the 
rank order based on post-program earnings. 
However, they did not match the earnings 
gains made by participants in other groups. 
As a result, their positions in the rank order 
based on post-program earnings was lower 
than their positions based on pre-program 
earnings. The most dramatic decline was for 
the group receiving "other" interventions. 
This group had the highest level of 
pre-program earnings, but the only group 
with lower post-program earnings was the 
group that received on]y remedial and basic 
educational training. 



For each participant group, the average 
earnings of employed terminees increased 
from the first to the fourth post-program 
quarter (Figure 6.13). The increases of each 
group were roughly proportioiuil. As a 
result, the rank ordering of the groups ba'^^ed 
on their fourth quarter earnings was very 
close to the order based on the first 
post-program quarter. 

Measures that reflect the earnings of 
employed terminees do not account for the 
different levels of unemployment of the 
groups in the post-program period. This 
study calculated the percentage of terminees 
from each group that was employed during 
each of the post-program quarters and 
examined the quarter-to-quarter changes in 
these rates. 

The post-program employment trends in 
Figure 6.14 indicate that there was a general 
decline in the percentage of terminees who 
were employed from the first to the fourth 
post-program quarter. However, there are 
some notable differences among the groups. 
Because employment increased in the early 
post-program period for the group 
receiving remedial and basic educational 
training, this was the only group that had a 
higher level of employment in the fourth 
post-program quarter than in the first. 
Between the third and fourth post-program 
quarters, employment dropped sharply 
among cn-the-job training and occupational 
classroom training participants. The 
magnitude of these declines are similar for 
the two groups, but much larger than the 
corresponding declines for other groups. 
The decline in employment for the 
on-the-job training group and the overall 
increase in employment for the remedial 
and basic group makes it appear that the 
employment rates for all groups are 
converging. However, we would need data 
for additional quarters to determine 
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Figure 5.13 

Post-Program Quarterly Average Earnings 
by Program Activity 
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Figure 6.14 



Post-Program Quarterly Employment Rates 
by Program Activity 



75 



-O 70 

> 
O 

1 S5 

ui 

4) 60 



55 



«0 



















EdCrt — 




Oth«r « * • 




.^1^ OJT — 




Multlpit » • 






N 

X 

X 




***** ^ 




• • • 




... _ 1 






• 

L-,^ 1 



2 3 

Post-Program Quarter 



ERIC 



65 



G3 



)TPA/UI Evaluation Study 



whether the convergence of these rates is 
real 

One way to view the earnings and 
employment trends in tandem is to 
construct an earnings measure that takes 
into account the zero earnings of the 
unemployed. In other words, this measure 
would use all terminees as the base for the 
calculation of average earnings instead of 
just employed terminees. 

When the quarter-to-quarter changes were 
examined using the alternative earnings 
measure (Figure 6.15), not all groups were 
found to have a continually increasing level 
of earnings. The average earnings increased 
from the first to the third post-program 
quarter for all six groups, but earnings 
declined for four of the groups between the 
third and fourth post-program quarters. 
The most dranuitic declines appeared for the 
on-the-job training and occupational 



classroom training participants. Earnings 
dropped for the groups receiving "other" 
interventions and multiple interventions, 
but the declines were almost imperceptible. 

These results suggest that the employment 
declines in the four groups were sufficiently 
large to offeet the earnings gains of those 
who were employed. Average eainings 
measured on the basis of those employed 
increased continually over the 
post-program period, but when the 
unemployed were included, the pattern 
reversed between the third and fourth 
quarter. However, the declines in 
employment of the other two groups did not 
have this effect. In order for this result to be 
obtained, the rate of decline in employment 
had to be large enough to offset the 
increasing level of earnings of those 
employed. 





Fbure 6.15 
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These results underscore the daiiger of 
using an average earnings measure based 
exclusively on those employed as an 
indicator of performance. Such an indicator 
can produce nusleading results. In one case, 
we found that two interventions had the 
same level of average earnings measured on 
the basis of employed terminees, but one of 
the interventions produced a markedly 
lower entered employment rate. Assuming 
that other factors are equal the intervention 
with the higher entered employment rate 
should be interpreted as the more effective 
of the two interventions. However, 
ignoring ihe zero earnings of those 
unemployed makes the two interventions 
look equally effective. 

Summary 

This chapter has illustrated the use of UI 
wagerecord data to examine the pre- and 
post-program employment and earnings of 
PY86 JTPA participants across various 
dimensions. The findings indicate the 
following: 

• Although the earnings of JTPA 
participants show a sharp ixUne 
prior to enrollment, employment 
levels remain fairly constant. 

• The average post-program earnings 
of terminees increase over the course 
of the post-program period, but 
employment levels decline. 

• Terminees with lower pre-program 
earnings and employment 
experiences tend to receive more 
intensive )TPA services. 



• The differentials found in the 
post-program earnings of 
participants receiving different 
types of interventions tend to 
correspond to the differences found 
in their pre-program earnings. 

• There is some evidence that remedial 
and basic education lead to increases 
in employment for participants 
which show up after a delay of a few 
quarters upon termination from this 
activity. 

Analyses using the expanded data sets of 
Phases II and III of this project will serve to 
clarify and extend these preliminary 
findings. 

Further research also should investigate 
whether or not the patterns found for all 
participants are the same within each 
demographic subgroup and isolate the 
reasons for the discrepancies. Such an 
analysis wouU increase the current level of 
understanding about the observed 
differences in performance among JTPA 
subgroups. Since these differences are the 
basis of performance standards 
adjustments, it would be interesting to 
assess what role program activities play in 
the determination of these adjustments. 

Research should focus too on the process 
by which JTPA participants are assigned to 
different activities. Although the evidence 
presented in this chapter suggests that 
pre-program employment and earnings are 
important considerations, it is likely that 
other criteria are also used. It is possible to 
use the information contained in this 
database to identify and evaluate some of 
these factors. 
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Endnotes 

1. DOL's program activity categories were collapsed to correspond to the categories used in 
this report. The data were obtained fron\ the U.S. Department of Labor, 1987. 

2. In Table 6.2, the entered employment rates produced by activity sequences received by 
less than 0.5% of all participants were suppressed in order to focus attention on the most 
common sequences. 
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Use of UI Data in the JTPA Performance 

Standards System 



One of the major features of JTPA that 
distinguishes it from earlier employment 
and training programs is its clear emphasis 
on program performance. The legislation 
stipulates that program funds are to be 
treated as an investment in human capital 
and requires JTPA programs to be evaluated 
on the basts of the return on this investment. 
The primary mechanism for evaluating the 
program is the JTPA performance standards 
system. 

At the core of the performiance standards 
system is d set of performance measures 
defined by the Secretary of the U.S. 
Department of Labor. For Program Year 
1989, there were twelve such measures 
specifically designed to reflect the multiple 
goals of JTPA. The Secretary is also 
responsible for assigning numerical levels of 
performance to each measure. These levels 
are defined as the national performance 
standards and are used as benchmarks for 
the national performance of the JTPA 
system. 

The JTPA legislation assigns the primary 
responsibility for implementing the 
performance standards system to the states. 
Although states may choose to set 
performance expectations for their SDAs to 
the levels defined by the national 
performance standards, they are 
encouraged to vary those standards. To 
assist states in setting performance 
expectations for SDAs, the Department of 



Labor (DOL) developed procedures, based 
on regression modeling techniques, for 
adjusting the national performance 
standards. These adjustments provide for 
factors that influence SDA performance 
directly but are considered beyond the 
effective control of the SDAs. 

The performance standards adjustment 
models are an innovative feature of the JTPA 
performance standards system. They zWow 
states to tailor the performance expectations 
of their SDAs to reflect the unique set of local 
conditions in SDA service areas. For 
example, performance standards models 
contain adjustments that lower the 
performance expectations of SDAs serving 
relatively high percentages of traditionally 
hard-to-serve client groups, e.g., welfare 
recipients, minorities, etc. In the absence of 
these models, each SDA would be expected 
to achieve the same level of performance, as 
defined by the national standards, 
regardless of the clientele it served. This 
would result in unreasonable perfonnance 
expectations for SDAs that provide services 
in areas that have high concentrations of 
hard-to-serve groups. 

UI wage-record data can play two 
important roles within the context of the 
JTPA perfonnance standards system. First, 
because UI data provide information on the 
pre-program employment of JTPA clients, it 
is possible to adjust performance standards 
based on the varying degrees of labor force 
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experience found among JTPA clients. 
Second, perfonnance measures based on 
post-program UI data can be used either to 
supplement or replace the current 
post-program measures as indicators of the 
long-term success of JTPA participants. 

The remainder of this chapter will be 
devoted to illustrating the ability of UI data 
to fill both of these roles through examples 
drawn from the eleven-state database. For 
example, UI data will be used to develop a 
set of potential perfonnance standards 
adjustment factors. These factors will then 
be tested within performance standards 
models to assess their effectiveness in 
explaining performance outcomes. Uldata 
will also be used to define a number of 
potential post-program performance 
measures, and several of these measures will 
be examined tn the context of a performance 
standards model. 

Peif ormance Sfandaids 
Adjustments 

Performance standards adjustment models 
convey strong policy messages to loca^ 
program operators. The mere presence of 
an adjustment for a particular hard-to-serve 
client group signals SDAs that they will be 
held harmless for their service to that group. 
Consequently, SDAs have a greater 
incentive to provide service to client groups 
represented in the models than to other 
groups. The models are thus a policy tool 
that can help direct JTPA services to 
hard-to-serve client groups. 

UI wage-record data can be used to define 
and identify client populations for which 
performance standards adjustments are 
desirable. One of the most significant 
barriers to employment that many JTPA 
participants face is their minimal level of 



labor force experience. This is evidenced by 
the important explanatory role the factor 
Tercent Unemployed 1 5 Weeks or More out ^ 
of the 26 Weeks Prior to Enrollment" 
(UNEMPL26) plays in nearly all of the 
performance standards models for PY89. 
Although the ability of this factor to explain 
SDA performance outcomes appears to be 
well-established, it is possible that 
additional factors or more effective 
alternatives to UNEMPL26 can be 
developed from pre-program UI 
wage-record data. At the very least, such an 
exercise will provide assurances that 
UNEMPL26 captures the most influential 
aspects of the pre-program labor market 
experiences of JTPA participants. 

The ai\alysis that follows represents a 
preliminary attempt to identify 
hard-to-serve client groups through UI 
wage-record data. For this analysis, we 
examined UI data for each participant to 
identify the pattern of employment during 
the pre-program period. We then calculated 
four potential adjustment factors on the 
basis of these patterns. The factors 
identified the participant as (1 ) unemployed 
during the first full quarter prior to 
enrollment (NOWORKQl ), (2) unemployed 
during the two quarters prior to enrollment 
(NOWORKQ2), (3) unemployed during the 
three quarters prior to enrollment 
(NOWORKQ3), or (4) unemployed during 
all four quarters prior to enrollment 
(NOWORKQ4). The four grovps of 
participants identified by these factors are 
not mutually exclusive. I^or example, 
participants who were unemployed during 
all four pre-program quarters would, by 
definition, be members of all fotu* groups. 
These four factors form a hierarchy of 
decreasing labor force attachment. 

The major problem with these Ul-based 
factors is that they cannot account for the 
possibility that a participant became 
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employed during the time 'ag between the 
start date for the quarter of JTPA enrollment 
and the actual date of enrollment. This can 
represent a substantial period, for example, 
when a participant enrolls during the last 
week of a fiscal quarter. Although it would 
be possible to examine the UI data to 
determine whether the participant was 
employed during the quarter oi enrollment, 
there would be problems in interpreting the 
information. In particular, it would be 
impossible to detennine whether UI wages 
reported for this quarter were the result of a 
job held prior to enrollment, after 
enrollment, or both. 

This drawback to using Ul-based factors 
would be eliminated if supplemental 
information collected by the states could be 
used to determine whether a participant was 
employed during this time period. For 
example, either the date of last employment 
or the number of weeks unemployed prior 
to enrollment could be used to ill this 
information gap. In the absence of this 
information, adjustment factors based solely 
on UI data will remain imperfect measures 
of pre-program employment history of 
JTPA clients. 

This imperfection in the Ul-based 
pre-program employment measures is not a 
major liability because the level of bias 
resulting from this measurement problem is 
likely to be small. It is doubtful that a 
substantial number of JTPA participants 
become employed in the period of time for 
which the Ul-based factors do not account. 
Furthermore, the impact of long-term 
unemployment that these factors reflect is 
unlikely to be substantially diminished by 
the recent employment of a small percentage 
of participants. Given the exploratory 
nature of this study, these conunents argue 
tha'i it is unnecessary to introduce 
procedures to correct this problem with the 
Ul-based measures prior to examining their 



'se as adjustment factors in performance 
standards models- 

To examine the ability of the Ul-based 
adjustment factors to explain JTPA 
outcomes, we developed performance 
standards models for two of the adult Title 
II- A performance measures, the adult 
entered employment rate and the adult 
average wage at placement. Since the 
number of SDAs represented on the 
database was insufficient to support the 
approach DOL used to construct the 
national models, we used an alternative 
methodology, the individual-level 
performance standards modeling approach, 
currently being used in Illinois and explored 
by other states. 

The individual-level performance 
standards modeling approach differs from 
the SDA-based approach DOL uses in that it 
focuses on the relationship between the 
factors and performance at the individual 
level rather than the SDA level. The two 
approaches provide different levels of 
adjustments and allow different 
interpretatiorw of those adjustments (see 
Trott et al., 1987, for details). However, since 
the relationships found at the individual 
level are the primary source of the 
relationships uncovered at the SDA level, 
the use of this alternative methodology is 
not expected to inhibit the ability to draw 
conclusions about the effectiveness of 
Ul-based adjustment factors. If Ul-based 
factors are found to be powerful predictors 
of performance at the individual level, they 
should also be effective in predicting 
performance at the SDA level. 

The procedures used to examine the 
impact ol using Ul-based adjustment factors 
in performance standards models involved 
several steps. First, we chose a set of 
explanatory factors to form a base model 
that would serve as a benchmark for 
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comparisTin. This base model, which we 
used for both performance measures, 
included six factors reflecting an array of 
client characteristics plus two county-based 
economic factors.^ These eight factors are 
listed in Table 7.1, which supplies 
descriptive statistics for the factors used in 
the models as well as the performance 
measures. 

We chose the explanatory factors solely on 
the basis of their past performance as 



adjustment factors. In other words, we 
made no attempt to examine all the potential 
factors contained in the eleven-state 
database in an effort to find the best set of 
factors. This task was clearly beyond the 
scope of this analysis where the goal is 
merely to demonstrate the use of Ul-based 
factors. 

The next step in this process was to 
calculate the factor weights (regression 
coefficients) using only the factors in the 



Table 7.1 



SDA-Level Statistics for the Variables Used in the Entered Employment Rate and 

Average Wage at Placement Models* 







Minimum 


Maximum 


Standard 


Factor 


Mean 


Value 


Value 


Deviation 


Female 


55.9 


36.9 


81.7 


7.7 


Black 


24.1 


0 


90.1 


22.9 


Hispanic 


3,9 


0 


41.7 


5.9 


Dropout 


26.0 


7.8 


54.8 


7.8 


Welfare Recipient 


22.8 


7.8 


51.1 


9,6 


Unemployed 15 of 26 Weeks 


45.9 


1.0 


79.3 


21.3 


County Employment Rate 


7.0 


2.3 


14.6 


12 


County Average Wage 


17.2 


11.4 


25.4 


2.8 


No UI Wages for 1 Quarter 










before Enrollment 


60.7 


43.6 


77.7 


6.6 


No UI Wages for 2 Quarters 










before Enrollment 


49.3 


31.3 


69.2 


7.0 


No UI Wages for 3 Quarters 










before Enrollment 


42.7 


24.9 


62.1 


7,1 


No UI Wages for 4 Quarters 










before Enrollment 


38.6 


23.2 


57.8 


6.8 


Adult Entered Employment Rate 


75.9 


55.2 


94.2 


8.5 


Adult Average Wage at Placement 


4.86 


3.92 


6.52 


.45 



* Utah SDAs are not repres^rted in the figures. 
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base model. After we obtained these 
weights, we introduced each Ul-based 
fiactor independently into the nnodel so that 
there would be only one such factor in the 
model at any given time. As factor entered 
the model, we re-estimated it to obtain a new 
set of factor weights. The resxilt of this 
process was five alterxuitive models for each 
of the two performance measures. Table 7.2 
shows the five models for the adult entered 



employment rate and Table 7.3 shows the 
models for the adult average wage at 
placement- 

The most surprising findii-^g from this 
aiuilysis was the poor performance of the 
UNEMPL26 factor in the base models for 
both performance measures. Although we 
anticipated that this factor would lose some 
of its explanatory power in the presence of 



Table 7.2 

Comparison of UI-Based Pre-Program Employment Factors 
Adult Entered Employment Rate Model 



Base Model Model Model Model 
Factor Model 12 3 4 



Female 
Black 
Hispanic 
Dropout 

Welfare Recipient 
Unemployec* 15 of 26 Weeks 
County Employment Rate 
County Average Wage 
No UI Wages for 1 Quarter 
before Enrollment 
No UI Wages for 2 Quarters 
before Enrollment -0.055 
No UI Wages for 3 Quarters 

before I2nrollment -0.060 
No UI Wages for 4 Quarters 

before Enrolbnent -0.062 

R-Square .03 .03 .03 .03 .04 

r** .51 .52 .52 .52 .52 

N for all Models 99074 



-0.026 


-0.027 


-0.026 


-0.025 


-0.026 


-0.046 


-0.047 


-0.047 


-0.047 


-0.047 


0.028 


0.028 


0.029 


0.029 


0.030 


-0.046 


-0.045 


-0.045 


-0.044 


-0.044 


-0.113 


-0.103 


-0.102 


-0.102 


-0.102 


*-0.001 


0.012 


0.009 


0.008 


0.007 


-0.936 


-0.898 


•0.8% 


-0.894 


-0.900 


-0.776 


-0.785 


-0.781 


-0.781 


-0.778 




-0.056 









* Not significant at alpha s .01 
Pearson correlation between the modeMerlved SDA performance expectation and actual performance. N - 118 
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Table 73 



Comparison of UI-Bascd Pre-Program Employment Factors 
Adult Average Wage at Placement Model 



Far tor 


Raft* 
Model 


1 


Mod«l 




4 


Female 


-.0072 


-.0072 


^.0072 


-.0072 


-.0072 


Black 


-.0034 


-.0035 


-.0035 


-.0035 


-.0035 


Hispanic 


-.0022 


'Om 


-.0022 


-.0021 


-.0021 


Dropout 


-.0049 


-.0049 


-.0049 


-.0049 


-.0049 


Wellare Recipient 


-.0010 


',0008 


-.0007 


-.0007 


-.0008 


I Tn«mn*?ivM] IS nf WmIta 

WltCUlwa^yCU X9 W< *9 VVCCIUI 












County Employment Rate 


,0133 


.0137 


,0137 


.0139 


.0136 


County Average Wage 


.0709 


.0706 


.0706 


.0706 


.0707 


No UI Wagee for 1 Quarter 












beloee Enrollment 




-.0012 








No UI Wages for 2 Quarters 












before Enrollment 






-.0017 






No UI Wages lor 3 Quarters 












before EnioUment 








-.0018 




No UI Wages for 4 Quarters 












before Enrollment 










-.0017 


R-Square 


.08 


J08 


JOS 


.08 


.08 


I** 


.71 


.72 


.72 


.72 


.72 


N for all Models » 99074 













* Not significant at alpha s .01 

** Pearaon cocrelation between the model-derived SPA perfomanoe expeci aaon and actual performance N « 118 



the Ul-based factors, we did not expect to 
be statistically iroignificant (alpha » .01/ in 
both of the base models. This result 
indicates that JTPA participants who were 
unemployed for 15 or more weeks of the 26 
weeks prior to their enrollment in the 
program had program outcomes that were 
similar to those of other participants. 

The mystery surrounding the poor 
performance of UNEMPL26 deepened once 
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we introduced Ul-based factors into the 
adult entered employment rate model. In 
the presence of Ul-based factors, 
UNEMPL26 passed ^he test of statistical 
significance, but to relationship to the 
entered efr.ployment rate was 
counterintuitive. One would expect that 
participants who were unemployed 15 or 
more weeks of the 2t weeks i. rior to 
enrollment would be less likely thai others 
to be employed at terminadon. However, 
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when Ul-based factors were placed in the 
model, the positive sign of factor weight for 
UNEMPL26 indicated that the)' were n\ore 
likely to be employed. 

An explanation of the effect of the Ul-based 
factors on UNEMPL26 is not easy to find. 
These results may be peculiar to the client 
population of the eleven states participating 
in this project. Unfortunately, the data 
needed to test this hypothesis do not exist, 
i.e., a national sample of JTPA participants 
with UI wage records attached. Until this 
mystery is solved, the reliability of 
UNEMPL26 as an adjustment factor in these 
models remains open to question. 

In contrast to UNEMPL26, the Ul-based 
factors performed exactly as expected. In 
the models for both performance measures, 
NOWORKQl had a strong negative 
relationship with the performance outcome 
under examination. This indicates that 
participants who were unemployed diiring 
the first full quarter prior to eiuollment were 
less likely to be employed at temuiuition and 
had lower wages when they were employed. 
Furthermore, each time we replaced a 
Ul-based factor with a factor that considered 
an additional quarter of imemployment, the 
magnitude of the weights tended to 
increase. This can be seen in Tables 7.2 and 
7.3, which show the change in the Ul-based 
factor weights as the criterion for 
unemploymentbecomesmorestringent,i.e., 
from NOWORKQl to NOWOKKQ4. The 
interpretation of the changes is that the 
longer participants were absent from the 
work force, the harder they were to serve. 

Given that each of the Ul-based factors was 
statistically significant and their 
relationship with the outcome measure was 
in the expected direction, the next step was 
to deternune the level of success of each 
factor in improving the ability of the model 
to predict SDA performance. To assess this 
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abUity, we calculated SDA performance 
expectations on the basis of the factor 
weights found in each model and correlated 
these expectations with actual SDA 
performance.^ If including the Ul-based 
factors in the base model substantially 
improved the ability to predict SDA 
performance, the correlation l)etween the 
SDA expectations and actual performance 
should be higher when the Ul-based factors 
are used to derive the expectatiorw than 
when they are not. 

The results of this analysis suggest that the 
Ul-based factors were only marginally 
successful in improving the ability of the 
model to predict SDA performance. 
Although SDA expectations derived from 
models including the Ul-based factors were 
more highly correlated with actual 
performance than the SDA expectations 
produced by the base model, the 
improvements were almost negligible. 
Further analysis revealed that most of the 
explatuitory factors could be removed from 
the models without causing severe loss of 
explanatory power. Table 7.4 shows the 
correlations that emerged each time we 
excluded a single factor from the base 
model. In the case of the entered 
employment rate model, only three factors 
(the indicator of welfare status and the two 
local economic factors) produced a 
substantial drop in the correlation once they 
were removed from the model. For the 
adult average wage at placement model, 
only the exclusion of the area average wage 
factor resulted in a substantially lower 
correlation.^ 

Using Ul-based pre-program 
unemployment factors in the performance 
staiKlards models is a matter of debate. On 
the one hand, it can be argued that since the 
factors do not appreciably increase the 
ability of the models to predict SDA 
performance, they should not be included in 
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Table 7.4 



Pearson Conelations Between Model Produced SDA Expectation and Actual SDA 
Performance After Excluding Factors 



Factor Excluded from 
Base Model 


Adult Entered Employment 
Rate Model 


Adult Average Wage at 
Placement Model 


Female 


.52 


.71 


Black 


.50 


.63 


Hispanic 


.52 


.71 


Dropout 


.51 


.67 


Welfare Recipient 


.45 


.71 


Unemployed 15 of 26 Weeks 


.51 


.71 


County Employment Rate 


.43 


.72 


County Average Wage 


.46 


.50 


Base Model 


.51 


.71 



N = 118 



the models. Those who view the models 
h-om a statistical perspective and focus on 
the use of the models to predict SDA 
performance will probably take this 
position. An alternative perspective 
considers the role of the models as a policy 
tool, i.e., their ability to direct attention 
toward hard-to-serve client groups. Given 
that Ul-based factors can identify and 
provide adjustments for hard-to-serve client 
populations, those with this perspective will 
argue that the factors should be included in 
the models despite their low explanatory 
power. 

Another issue in this debate focuses on the 
concern for equity in the performance 
standards system. Although Ul-based 
unemployment factors may not improve the 
overall ability of the modeb to predict SDA 
performance, they can still affect the 
interpretation of how particular SDAs 
perform. In faci, the use of these factors can 



have a substantial impact on the 
performance expectations of those SDAs 
that serve a relatively high or low 
percentage of the long-term unemployed. 
For example, an SDA that was previoujily 
designated as a poor performer may not be 
interpreted as such after its performance 
expectations are adjusted on the basis of its 
level of service to the long-term 
unemployed. This reinterpretation may 
happen in only a few SDAs, but Ul-based 
pre-program unemployment factors can 
nevertheless promote a significant degree of 
equity in the assessment process. 

Developing pre-program earnings factors 
is much more complex than constructing 
pre-program employment factors. A 
pi'e-program earnings factor should reflect 
the stock of human rapital that participants 
bring into the program. However, it is 
difficult to identify a single pre-program 
quarter where the earnings are an adequate 
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measure of the stock of each participant. 
The well-documented pre-program 
earnings dip among many JTPA 
participants illustrates this problem. 
Bamow (1987) notes that the decline in 
earnings for the quarters immediately 
preceding program participation is either a 
transitory phenomenon or the result of a 
permanent decline in human capital If the 
dip is a transitory phenomenon^ then the 
pre-program earnings factor should be 
based on wages received during a more 
stable period in the participant's 
employment history. This argues that UI 
wages for at least the third or fourth quarter 
prior to enrollment should be used as the 
indicator for a paitidpant's stock in human 
capital On the other hand, if the dip 
indicates a permanent decline in human 
capital, the factor should be based on wages 
received in the period immediately prior to 
program participation. 

The unsettling aspect of this problem is that 
the decline in pre-program earnings is 
transitory for some JTPA participants and 
permanent for others. This is especially true 
in the Title III program since the skills of 
some dislocated workers are still in demand 
in the labor market, but the skills of others 
are devalued or obsolete. The ideal solution 
to this problem is to identify the decline as 
either transitory or permanent and adjust 
the calculation of the factor accordingly. 
However, it is extremely difficult to make 
this identification because both types of 
decline look exactly the same. This poses a 
major obstacle to developing performance 
standards adjustment factors based on 
pre-program earnings. 

There are also technical issues that must be 
addressed in constructing a Ul-based 
pre-program earnings factor regardless of 
how this conceptual issue is resolved. For 
example, severance pay and other 
lump-sum payments will inflate the wages 



reported in the quarter that a participant 
terminates from a job. As a result, complex 
decision rules must be developed and 
invoked if the pre-program earnings factor 
is adequately to measure the stock of human 
capital that participants bring into the 
program A related issue is whether zero 
earnings should be used in the calculation of 
the factor. For example, suppose the 
decision was made that wages for the third 
full quarter prior to enrollment were the 
appropriate focus for this factor. The next 
issue that must be addressed is whether the 
factor should represent the average earnings 
for all partkipants, the employed and the 
unemployed. 

Given the complex issues that must be 
resolved prior to developing a Ul-based 
pre-program earnings measure, v:^ made no 
attempt to construct such a measure for this 
report. However, the near certainty that 
pre-program earnings factors will be 
effective, if not the best, predictors of 
program outcomes suggests that a thorough 
examination of these issues should be given 
a high priority in future research. 

Peiformance Standards 
Measures 

The second role U! wage-record data can 
play in the performance standards system is 
in the development of post-program 
performance measures. This section will 
consider two types of Ulbased 
post-program measures: (1) earnings 
measures based solely on VI wage-record 
data, and (2^ employment measures, 
including employment retention measures 
developed from both termination data and 
Ul-based data. Many of the statistical 
analyses will focus at the individual level, 
and we will formulate and examine the 
measures at this level. 
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A key consideration in the design of 
post-program performance standards is the 
length of the post-program period. States 
use the national performance standards to 
set the performance expectations for their 
SDAs. These expectations are then used to 
judge the actual performance of SDAs for 
the purpose of awarding incentive grants 
and issuing sanctions. Studies iiKiicate that 
the longer the post-program period, the 
more likely the corresponcing measures 
will reflect the net impact of the program. 
However, ihere are difficulties in 
administering a performance management 
system when the point of assessment is too 
far removed from the provision of services- 
If the post-program period is too long, 
incentive grants to SDAs may be based on a 
level of performance that has since 
deteriorated, or sanctions may be imposed 
after services have improved. From a 
program management prospective, it is 
desirable to minimize the length of the 
post-program period. 

The importance of these management 
issues can be seen in the decisions that were 
made in the development of the current 
post-program performance management 
system. Despite the fact that 26-week 
post-program measures were more highly 
correlated with net impact benchmarks, the 
decision was made to use a 13-week 
post-program period (Zomitsky, Bloom, 
Schneider, and Sharick, 1985). This decision 
was based, in part, on the desire to minimize 
the time lag between the period of actual 
performance and the beginning of the 
assessment process. In other words, slightly 
less effective indicators of long-term 
impacts were selected in order to facilitate 
the implementation of the measures. 

Post-program performance data must not 
reflect employment while a participant was 
in the program. Therefore, the only 
Ul-based post-program measures that can 



be produced in a time frame comparable to 
that of the cuirent measures are those that 
focus on information reported for the 
quarter after the quarter of termination. 
Since most states indicate a reporting lag of 
one quarter for their UZ data, it will take at 
least six months from the close of a program 
year before one full quarter of post-program 
data is avail9ble for each participant who 
terminated during that program year. This 
time lag is only three months longer than the 
time it takes to generate survey data. 
However, for each subsequent 
post-program quarter, this time lag must be 
extended by three months. These added 
delays will rush *he corresponding 
assessments farther and farther away from 
the provision of services. Therefore, 
Ul-based post-p: ogram measures focusing 
on the first full post-program quarter should 
be viewed as the primary alternative to the 
current survey based measures, 

EARNINGS MEASURES 

Post-program earnings measures based on 
UI data are calculated with a longer 
measurement interval than those based on 
survey data. Because the survey-based 
measure focuses on earnings during the 
thirteenth week of the post-program period, 
it requires participants to be employed only 
during that week. Information is lost for 
participants who had earnings at some point 
during the post-program period, but not 
during the thirteenth week. Furthermore, 
the survey-based measure does not reflect 
earnings that participants had in any other 
week, even if they had earnings in the 
thirteenth week. Other measures must be 
used to reflect the duration of employment 
over the post-program ppriod. 

The interval used to calculate 
post-program earnings measures based on 
UI data reflects earnings over the entire 
post-program quarter. Tlie longer interval 
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thus means that the measure gives a truer 
picture of the post-program eamiTigs of 
participants. The measure also provides a 
better picture of the duration of employment 
because the level of eamings in a quarter is 
Influenced by the length of time participants 
were employed in the quarter. This feature 
can be further enhanced if Ul-based 
measures include the zero eamings of those 
who were not employed in the 
post-program quarter. 

Studies focusing on the relative validity of 
various post-program me - ^urps clearly 
demonstrate the importance of capturing 
the duration of employment. For example, 
Bloom and McLaughlin (1982) noted that 
earnings gains among job training 
participants were primarily a function of the 
length of time that the participants worked 
in the post-program period, not the hourly 
wage rate. The validation tests of Zomitsky 
et al. (^985) supported this conclusion. The 
measures that correlated most highly with 
the validation benchmarks were those that 
reflected the amount of time worked in the 
post-program period. 

Although it can be argued that the duration 
of employment should be measured 
independently of earnings because it is 
conceptually distinct, there are advantages 
to capturing it with an earnings measure. 
Combining the two reduces the number of 
performance measures and thus streamlines 
the performance standards system. States 
and SDAs are likely to find this option 
attractive since it will simplify program 
mpaagement. 

To illustrate the iinpact of the duration of 
employment on the interpretation of the 
eamings data, we constructed two sets of 
post-program eamings measures based on 
the total UI wages participants received 
during a given post-program quarter. The 
first set excluded nonmatched participants 



from the calculation of the measures, thus 
reflecting the average eamings only of those 
who were employed during a given 
post-program quarter. The second set 
included nonmatched participants in the 
calculation by assuming zero eamings for 
those participants. The second set thus 
provided the average earnings of all 
terminees in a given post-program quarter, 
not just those who were employed. 

Table 7.5 compares the results. The 
different treatments of nonmatched 
participants had a dramatic impact on the 
information conveye ■ by the measures. The 
measures that include the unemployed 
participants depict a substantially lower 
level of eamings than those based solely on 
the eamings of employed participants. This 
result should be expected. The two 
measures are calculated using the same level 
of total eamings but a different population 
base, i.e., employed terminees versus total 
terminees. 

The two sets of measures produce dif fei ent 
eamings patterns. The set that includes only 
employed participants shows average 
eamii gs increasing over time. The set that 
includes the unemployed as well as the 
employed show average eamings declining 
in the third and fourth post-program 
quarters. Post-program employment 
patterns explain this apparent contradiction. 
The number of employed participants 
consistently declines over the course of the 
post-program period. For example, the 
percentage of participants who were ever 
employed during the post-program period 
declined from a high of 67.1% during the 
first full quarter to a low of 57.4% for the 
fourth. Over half of the decrease occurred 
between the third and fourth quarters. 
Despite the fact that those who were 
employed in later quarters received higher 
wages, the declining level of employment 
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Table 7.5 



Comparison of Alternative AvcragjEamings Measures 



Post-Program Quarter 


Average Earnings Based on . 


Employed Terminees 


All Terminees 


First 


1993 


655 


Second 


2143 


786 


Third 


2231 


835 


Fourth 


2350 


1003 



N = 99074 



depressed the level of average earnings 
when it was measured for all participants. 

The two measures are complementary 
rather than contradictory. Each provides a 
perspective on the post-program earnings of 
JTPA participants that is not apparent from 
the other. The two together provide a better 
und'^rstanding of the dynamics underlying 
the earnings of participants. 

Fast studies have consistently found tliat 
earnings measures based on all participants 
indicate long-term program impacts better 
than measures that focus exclusively on 
employed participants. The net impact 
benchmarks used in these studies are based 
on the experience of all participants. Since 
earnings measures confined to the 
experiences of employed participants 
cannot capture the impacts of the program 
for those who are unemployed in the 
post-program period, they are relatively 
poor indicators of these benchmarks. In 
contrast, earnings measures based on all 
participants are typically the best indicators 
of long-term program impacts. 

In order to assess how adjustment factors 
react to post-program earnings indicators 
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measured at different points in the 
post-program period, we examined models 
for the earnings measures using all 
participants as the population base. For this 
analysis^ we excluded the two economic 
factors from the models because of the 
conceptual problems involved in using 
economic conditions measured at one time 
point to explain outcomes measured at 
different time points. 

The results of this analysis reveal that the 
effects of client characteristics change as the 
focus shifts from the first to the fourth 
post-program quarter (Table 7.6). For 
example, the negative weights for females, 
blacks, and dropouts tend to increase, but 
the positive weight for Hispanics cind the 
negative weight for welfare recipients tend 
to decrease. The most interesting change is 
the decline in the weight for the welfare 
factor. Although the earnings difference 
between v/elfare recipients and 
nonrecipients remains statistically 
significant in all quarters, the decline in the 
weight of the factor indicates that there is 
less of a difference in the later post-program 
quarters. 
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Table 7.6 

Comparison of Models for Various UI-Based Post-Program Average Earnings Measures 



Average Earnings Based on All Terminiecs 





Post-Program Quarter 




Factor 


1 


2 


3 


4 


Female 


A OR 


-3 14 


-3.26 


-3.55 


Black 


-1.66 


-1.66 


-1.70 


-2.05 


Hispanic 


1.74 


2.06 


1.63 


0.85 


Dropout 


-3.29 


-3.74 


-3.82 


-3.82 


Welfare Recipient 


-1.97 


-1.85 


-1.83 


-1.33 


Unemployed 15 of 26 Weeks 


-0.38 


--0.25 


*-0.19 


0.87 


No UI Wages for 1 Quarter before Enrollment 


-3.93 


-4.28 


-4.37 


-4.22 


R.Square 


.05 


.05 


.05 


.04 


r** 


.51 


.42 


.42 


.41 


|N for all Models -99074 











* Not significant at alpha = .01 
Pearson correlation between the model-derived SDA perfonnanoe expectiL >n and actual performance. 



The weights for the pre-program 
unemployment factors also change, but not 
in the same direction. The weights for the 
UT-based measure tend to become more 
negative (the expected direction). The 
MIS-t>ased measure has a negative weight in 
the first full post-program quarter and a 
positive weight in the fourth quarter, both of 
which are statistically significant 

Despite these changes, the relative effects 
of the factors within each of the models is 
remarkably similar. In other words, if the 
factors are ordered on the basis of the 
direction and magnitude of their weights, 
the resulting rankings of the factors are 
nearly identical across the models. This 
indicates that the changes in weights did not 
have a major 'mpact on the relative 



importance of the factors in explaining 
post-program earnings measured at 
different time points. 

To examine the ability of this set of factors 
to explain post-program outcomes, we used 
the four sets of factor weights to produce 
SDA post-program performance 
expectations for each of the post-program 
quarters. We then correlated these 
expectations with summary measures 
reflecting actual performance at the SDA 
level. These correlations indicate the 
explanatory power of the models at the SDA 
level. 

The ability of this set of adj^tment factors 
to predict post-program outcomes declines 
sharply after the first post-program quarter. 
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Although the correlation between the 
model-derived expectations derived from 
the model and the actual outcomes for the 
first post-program quarter was .51, the 
correlation for the second quarter declined 
to .42 and stabilized at that level in 
subsequent quarters. This suggests that the 
program outcomes are less predictable 
when they are measured at later stages of the 
post-program period. 

EMPLOYMENT MEASURES 

The UI wage record data can also be used 
to develop a variety of post-program 
employment measures, each providing a 
different perspective on the post-program 
experiences of JTPA participants. The 
sections that follow describe a number of 
these measures and discuss their major 
advantages and disadvantages. This 
discussion illustrates the wide range of 
options for constructing Ul-based 
post-program employment measures. 

Quarterly Employment Rates. The first set 
of potential measures are the easiest to 
understand and construct. These rates 
reflect the percentage of terminees who 
worked in Ul-cove ed jobs during a given 
post-program period. The number of 
terminees employed during the 
post-program period is divided by the total 
number of terminees: 

• TtrmiHtea Employtd During Po$t-Program Period 
Total TerminttB 

The post-program period used in these 
measures can be defined using one or more 
quarters of post-program data. 

The major disadvantage of these measures 
is that participants employed for a short 
time have the sama impact on the 
employment rate as those employed for a 
longer time. Since the only criterion is 



nonzero UI wages, participants who are 
employed for only one day over the 1 3-week 
quarter will be included iii the calculation of 
the rate. Thus, these measures are not very 
effective indicators of the long-term success 
of participants. 

CXiarterlv Employment Rates Constrained 
by a Minimal Earnings Expectation. These 
rates are constructed in a similar manner, 
but they include an additional criterion to 
ensure that employment during the quarter 
reflects some minimal duration. The 
criterion is a predefined earnings threshold, 
which could be set in a variety of ways. For 
example, it could be set to correspond to the 
earnings level that would result from 20 
hours of employment at a minimum wage 
job for each week of the quarter. To calculate 
the rate, the number of terminees employed 
during quarter i with total wages greater 
than the minimal earnings constraint is 
divided by the total number of terminees. 
These rates are more effective indicators of 
post-program performance because they 
reflect the percentage of participants with 
meaningful levels of employment in the 
post-program period. 

The major disadvantage of these rates is 
that setting the minimal earnings 
expectation is an arbitrary process that can 
be a source of considerable debate. Since 
there are no guidelines available for 
determining the earnings expectation, a set 
of criteria will have tobe developed to assess 
the various alternatives. The time involved 
will delay the implementation of the 
measures. 

Employment Retention Rates. These 
measures are designed to determine the 
success of JTPA participants in retairung 
employment in the post-program period. 
There are two major options for calculating 
these rates. The first option looks at the 
number of terminees employed at 
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termination and during a given 
post-program period as a percentage of the 
number of terminees employed at 
termination. The second option looks at the 
number of terminees employed at 
termination and during a given 
post-program period as a percentage of all 
terminees. The t wo options are expressed in 
the follov^ing equations: 

Option I 

« TtrmiHee$ Emplm/ed at TermiHation 
and durinjf tht Poat-Program Period 
i TtrmiHttM Employed «t Termination 

Option 2 

# TermineeB Employed Mt Termination 
and during the Pott-Program Period 
Total Termineee 

The post-program period can be defined as 
including one to four quarters. If more than 
one quarter is used, the rates can incorporate 
a more stringent criterion of employment. 
For example, the definition of a two-quarter 
period requires that the participant be 
employed in both quarters. 

Retention rates as a performance measure 
have a major advantage. They correspond 
more closely with the intention of the JTPA 
legislation, which identifies "retention in 
unsubsidized employment" as an 
appropriate measure of the performance of 
JTPA programs. Neither the quarterly 
employment rates nor the current 
employment measures based on survey data 
address retention. None impose the 
requirement that the terminee be employed 
at the time of termination from the program. 

On the other hand, retention rate measures 
fail to capture the duration of employment 
in the post-program period. Terminees 
employed for only one day will have the 



same impact on the retention rate as those 
who are employed for the entire period. 

A second disadvantage of these measures 
is that they do not capture the employment 
of participants who gained employment 
after termination. This could be a se/ious 
problem because some interventioiw may 
have a delayed effect. For example, this 
study found that participants receiving 
remedial and basic educational training 
have higher employment races in later 
post-program quarters than in the first. If 
this is due to the delayed effects of these 
interventions, it can be argued that retention 
raves are biased against them. 

Emplovment Retention Rates Constrained 
bv a Minimal Eamines Expectation. These 
performance measures have all the 
advantages of the unconstrained retention 
measures and the inclusion of a minimal 
earnings expectation to capture 
employment duration. However, they still 
ignore the possibility of the delayed effects 
of program interventions and the problems 
associated with defining the minimal 
earnings expectation. 

lob Retention Rates. Job retention rates 
focus on whether participants retain the 
same job they obtained at placement. These 
measures are possible because UI wage 
records contain an employer identification 
number. There arr ^wo ways these rates can 
be calculated: 

Option 1 

§ Termineet Employed by tiie tmployer at 
Termination and during the Poat-Progra-n Period 
« Terminee* Employed at Termination 



Option 2 

• Terminees Employed by the Employer at 
Termination during the Pott-Program Period 
Total Termineet 




1 .'i ' 
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Job retention rates capture the quality of job 
placements. The implicit assumption is that 
if the program places participants into 
meaningful jobs, they are likely to retain 
these jobs over the post-program period. 
Thus, job retention rates measure the quality 
of the placements more directly than 
employment retention rates. 

A number of technical barriers make it 
difficult to implement job retention 
measures. For example, there is the problem 
of determining the employer identification 
number of the employer at termination. 
States may not record this infomuition/ or 
they may not recora it accurately. 
Furthemiore, since employers are permitted 
to change their identification numbers 
under certain conditions, it can sometimes 
only appear that the terminee changed jobs 
when in fact, the employer changed 
numbers. 

In addition, there are conceptual concerns 
with using job retention measures. Such 
measures assume that terminees moved 
from the job in which they were placed at 
termination because of the poor quality of 
the job. However, the job may have 
provided the opportunity for advancement. 
Movement could thus be perceived as 
positive. 

Cumulative Employment Rates. 
Cumulative employment rafss attempt to 
capture the advantages of employment 
retention measures and, at the same time, 
provide credit for terminees who obtain 
employment after termination. These rates 
are calculated in two steps. The first step is 
to determine the ntmiber of terminees who 
were employed at termination and for each 
post-program quarter and to divide the sum 
by the number of observation periods. The 
result is the average number of employed 
terminees per period. In the second step, 
this average is divided by the total number 



of terminees. Suppose the rate is based on 
two quarters of post-program employment. 
The rate would be calculated as follows: 

(• Temdnett Cmfloytd at TerminatioH + * TtrmineeB 
Employed in Ou Pint Pott-Pngnm Qiutrttr ♦ 
i Termnue$ Employtd in ttu Second 
Peit'Pngr*m Quarter) 1 3 
ToM Termineea 

This rate will range from 0 to 1 . A value of 
1 indicates that all participants were 
employed at termination and at each 
posi-program quarter. A mininial earnings 
expectation can be applied to strengthen the 
ability of the measure to capture 
employment duration. 

The cumulative employment rate provides 
credit for employment at termination and at 
each stage in the post-program period 
without requiring that the terminee be 
employed at termination. Although it is not 
strictly an employment retention measure, 
the value of the rate will increase in relation 
to the percentage of participants who retain 
their employment. However, the rate is also 
influenced by increases in the number of 
terminees who gain employment after 
tern\ination. 

The major disadvantage of this measure is 
that it is difficult to explain and represents a 
departure from the way performance has 
traditionally been measured. If it is 
employed as a standard, substantial 
technical assistance may be required in 
order to familiarize states and SDAs with its 
properties so that they will feel comfortable 
with its use in the performance management 
system. 

A second disadvantage is that the measure 
assumes that employment gained after 
termination is a function of program 
interventions. This may be true for some 
interventions, but it may not be true for 
others. As a result, the measure may 



r2 

84 



Chapter Seven 



inappropriately reward some SDAs on the 
basis of performance levels that were not a 
product of program interventions. 

The above examples indicate some of the 
options that can he explored with Ul wage 
record data. The issues that remain to be 
addressed are the policy implications of 
these choices. Although this cursory 
exanunation of the Ul-based measures has 
exposed some of these implications/ a more 
focused analysis needs to be performed. 
Such an analysis will benefit from the 
observations and comments of program 
agents operating at various levels of the 
JTPA system. 

Summary 



The evidence presented in this chapter 
illustrates that UI wage-record data can play 



two important roles in the context of the 
JTPA performance^standard system. First, it 
can be used to' identify groups of 
participants with limiteoremployment 
experience so that performance standards 
can be adjusted on the basis of the level of 
service to these hard-to-serve client groups. 
SDAs that target services to these groups 
w'll find such adjustments a welcome 
addition to the current models. 

UI wage-reco*xi data can also be used to 
develop potential performance standards 
meeisures. Such measures will correspond 
more closely to the 'ntent of the JTPA 
legislation than the current measures based 
on survey data. Furthermore, evidence 
from past studies suggests that these Ul 
based measures will be more powerful 
indicators of long-term program impacts. 
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Endnotes 



1 . The use of the county-based economic indicators resulted in the exclusion of one state fron\ 
the analytical database. This state, Utah, did not supply the clearinghouse with data identi- 
fying the county of residence for its participants. 

2. SDA perfomumce expectations were calculated by applying the same procedures used in 
conjunction with DOL perfomwince standards models. For a description of how this is 
accomplished using factor weights derived from individual level data, see Baj and Trott, 1988. 

3. It may surprise some that the exclusion of the factor relating to the gender of the participant 
did not significantly decrease the correbtion for the average wage at placement given the 
strong effect of this factor in the base model. Although the results are not provided, we fouini 
that the effects of this factor were largely transmitted though other factors in the model, most 
notably welfare status. 
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Summary of Major Findings and Implications 



This report has conveyed the results of the 
first phase of a feasibility and demonstration 
project. The task has been to bring together 
JTPA terminee records with UI wage 
records and assess the potential for using UI 
wage records to monitor the pre- and 
post-employment and earnings experiencei 
of JTPA participants. Subsequent phases of 
this project will increase the number of states 
and add data on ternunees from Program 
Years 1987 and 1988. Analyses on these 
expanded data sets should serve to clarify 
and extend the findings summarized below. 

Major Findings 



THE UNEMPLOYMENT INSURANCE 
WAGE-RECORD DATA 
COLLECTION SYSTEM 

The UI system had its beginnings in the 
Federal Unemployment Tax Act (FUTA) of 
1937. The implementation of FUTA was 
influenced by the federal offer of an offset or 
credit against the tax for states meeting 
federal standards of implementation. The 
U ficit Reduction Act of 1984 increased the 
level of uniformity among state systems for 
collecting UI data. Before the legislation, 
some states had adopted a wage request 
approach to operate their UI systems. 
Employers in these states submitted wage 
information only when an unemployment 
insurance claim Iiad to be processed. As of 
Septembei 30, 1988, employers in all states 
were required to submit quarterly wage 
reports. The Deficit Reduction Act has thus 



ERIC 



served to eliminate the wage request option 
for states. 

Technical discussions of the potential use 
of UI wage records for longitudinal analysis 
typically center on four topics: coverage, 
accuracy, timeliness, and confidentially. 
Virtually all for-profit, private-sector 
businesses are liable for UI taxes. Therefore, 
they are covered by state UI systems. 
Although states use different payroll and 
employment thresholds, all must meet 
FUTA standards to remain eligible for the 
federal tax credit. Over 90 percent of all 
employment is covered and reported to the 
UI system. 

There are several reasons to expect a high 
degree of accuracy for all data elenr\ents 
contained in state UI files. Accuracy is 
especially high for earnings reports and is 
improving. However, there is a problem 
with using UI information to determine the 
location (county) of employment because of 
the reporting practices of 
multiestablishment firms operating in a 
state. Furthermore, these practices are not 
uniform among the states. 

Tax liable employers must report covered 
workers and their earnings to the state 
within 30 days of the end of each quarter. 
Although late reporting occurs, the 
incidence is typically low because it is 
subject io penalty and interest charges. 
Small businesses are more likely to report 
late than large businesses. However, this 
sitiiation is expected to improve as more 
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small businesses adopt electronic reporting 
procedures. 

Use of UI wage records for evaluation 
purposes must satisfy concerns for 
confidentiality. State unemployment 
statutes contain a confidentiality provision 
protecting the anonymity of individual 
employees and reporting employers. 
Release of UI wage-records normally 
involves an agreement between the state 
employment security agency and the 
receiving party. The agreement assures that 
the integrity of the information will be 
maintained and that employees and 
employers will not be identified. 
Additionally, many procedures to encrypt 
wage records exist so that individual social 
security numbers and employer 
identification numbers are not revealed. 
Many examples exist in which third-party 
users have maintained the integrity of the 
wage-record information. This project 
involving 11 states is but one example of 
cooperation among state agencies and 
successful release of UI records without 
violations of confidentiality. 

The major findings of this examination of 
the UI wage-record reporting system are 
summarized as follows: 

• Virtually all jobs that most observers 
would cor\sider appropriate targets 
for JTPA terminee placement are 
covered by the UI reporting sv^iem. 

• Nearly all of these UI covered jobs 
offer n\oney wages, commissions, 
and bonus forms of compensation, 
which are subject to accurate 
reporting practices. 

• The incidence oflate reporting is low 
and expected to improve as more 
businesses adopt electronic 
reporting procedures. 



* State laws and administrative 
regulations need not act as a barrier 
to the responsible use of the 
wagerecord data in full compliance 
with the spirit and case law of state 
privacy requirements. 

THE UNEMPLOYMENT INSURANCE 
WAGE-RECORD SYSTEM AS AN 
ASSESSMENT TOOL 

Because eacl. UI quarterly wage record 
contains information from a single employer 
for a single employee, the system allows 
program evaluators a great deal of 
flexibilit) . When JTPA records are merged 
with UI wage records, post-JTPA program 
earnings histories can be examined by 
participant characteristics (e.g., age, sex, 
race), types of program interventions (e.g., 
basic educational training, on-the-job 
training), service provider, geography 
(county of residence, SDA, state), industry 
of pre-program employment, industry of 
post-program employment, or any 
combination of the above. 

Perhaps the greatest value in linking the 
JTPA and UI records li^s in the opportunity 
to focus analyses at the individual level. 
Individual-level ai\alyses provide insights 
into the labor-force experiences of JTPA 
participants that simply cannot be obtained 
from summary data such as the data 
currently collected through the JTPA annual 
status report. Furthermore, these analyses 
can be conducted without jeopardizing the 
confidentiality of clients or businesses. 

Another distinct advantage of quarterly UI 
wage records as an assessment tool is that 
they permit the tracking of post-program 
experiences well beyond the current 
thirteenth- week post-program period. By 
continually appending additional quarters 
of wage information to client records, one 
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can follow the progress of JTPA terminees 
for several years. If desired. The use of the 
UI wage records could also substantially 
expand the pre-program information base 
on JTPA clients without imposing the 
burden of collecting additional data on local 
program operators. 

The UI wagerecord data are an excellent 
resource for studies concerned with 
assessing the net impact of JTPA and other 
employment and training programs. Since 
UI data can be obtained both for program 
participants and members of the 
comparison (or control) group, the 
employment and earnings outcomes of the 
two groups can be deHned and measured 
consistently. Finally, UI wagerecord data 
are an extremely cost-effective alternative to 
other sources of post-program information. 

This review of features of the UI wage 
record system as an assessment tool 
revealed the following: 

• The use of UI wage-record data 
would enable detailed analyses of 
the long-term employment and 
earnings patterns of JTPA terminees. 

• UI wage-record data offer great 
promise as a resource for studies 
designed to measure the net impact 
of employment and training 
programs. 

• UI wage-record data are a more 
cost-effective means for obtaining 
post-program employment and 
earnings information than surveys 
of participants. 



The Development of the 
U-State Database 



One of the major activities of the first phase 
of this project involved the construction of a 
database containing longitudinal UI 
wage-record data and JTPA program data 
for the 11 states participating in the project. 
The effort required each state to negotiate 
agreements for the exchange of data within 
the state and with the Commission. Each 
state also liad to match the JTPA and UI files 
and produce a set of standard tables. The 
clearinghouse was required to establish data 
transfer conventions for all states to follow, 
compile state files, check them against the 
standard tables the states produced, and 
merge the state files into a single file. Each 
of these steps was accomplished in a timely 
and efficient manner. Most of the problerrw 
were encountered in processing the JTPA 
files rather than the UI files. However, these 
problems were minor and, once identified, 
were quickly corrected. 

The project paid particular attention to the 
activity or intervention records maintained 
in the state JTPA management information 
systems. Activity data were particularly 
prone to coding errors, and there is the 
suspicion that SDAs tend to underreport 
this information. 

The experience of constructing the 1 1 -state 
database of matched UI wage records and 
JTPA terminee and program intervention 
records is summarized as follows: 

• The technical problems of matching 
records from the two data systems 
are reasonably minor and easily 
overcome. 
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• If a state resolves to use the UI 
wage-record information as an 
assessment tool, necessary 
agreements and procedures for 
exchanging data can be established 
and executed. 

• There are a number of specific issues 
regarding the accuracy and 
reliability of program activity data. 

COMPARISON BETWEEN UI 
WAGE-RECORD DATA AND THE 
POST-PROGRAM DATA OBTAINED 
FROM SURVEYING JTPA 
PARTICIPANTS 

The database constructed by this project 
provided an opportunity to assess some of 
the issues of data quality surrounding the 
post-program survey. First, the study 
examined response rates by subgroups 
using the individual-level data from the 
JTPA records. Certain subgroups of the 
JTPA population had lower response rates 
even after we controlled for the effects of 
employment status at termination. In 
particular, males, minorities, high school 
dropouts, and welfare recipients had 
significantly lower response rates to the 
survey. This evidence suggests that the 
level of service an SDA provides to these 
groups influences the response rates it 
achieves on the post-program survey. 

The project's database also provided the 
first large-scale opportunity to ex'imine the 
issue of nonresponse bias and its impact on 
the estimates of program performance. We 
found that the post-program employment 
experiences of survey respondents and 
nonrespondents were distinctly different. 
In particular, respondents had higher 
post-program employment rates than 
nonrespondents, suggesting that the survey 
estimates of post-program employment are 



inflated. Further analysis revealed the 
source of this bias. It was determined that 
the current nonresponse adjustment 
procedures were ineffective in combatting 
this problem. 

Overall, there was a fairly high degree of 
correspoiulence between the employment 
information obtained from respondents to 
the survey and their UI records. Efforts 
were made to identify the source of the 
discrepancies between the two data sources. 
Although data Umitations inhibited this 
analysis, the evidence suggests that two 
explanations are likely to account for the 
majority of these mismatches, namely, 
out-of-state employment and noncovered 
employment, e.g., self-employment. In both 
cases, the problenns associated with these 
coverage issues can be adequately 
addressed through a variety of remedial 
measures. 

The relative costs of the two data collection 
options were examined. The study 
estimated that if each state in the nation 
implemented a Ul-based data collection 
system, the total start-up cost would be less 
than one-half of the current cost of collecting 
post-program survey data. Furthermore, 
the ongoing costs would be less than 
one-fifth the cost of participant contact. This 
would result in a saving of over two million 
dollars each year. 

Another factor to consider in this 
comparison is the quantity of data 
purchased with these funds. The 
post-program survey principally offers one 
week's worth of employment and earnings 
information on a sample of one year's 
terminees. In contrast, UI data provides 
multiple quarters of pre- and post-program 
data for the universe of those termituited 
from the program in a gi 'en year plus 
additional post-program information for 
terminees from previous years. 
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Finally, survey costs can be expected to 
increase over time. Since survey research is 
labor intensive, the cost of conducting 
surveys is linked to the prevailing wage rate. 
As a result, costs are Ukely to increase in the 
future with the rise in the minimum wage. 
In contrast, the costs involved in usmg UI 
wage-record data depend on technology. 
As a result, they should rennain reasonably 
stable and may even decline in response to 
future advancements in data processing 
technologies. 

The major findings of this comparison of 
the two data sources are as follows: 

• There is evidence of nonresponse 
bias in the 13- week post-program 
survey data that inflates the 
estimates of post-program 
performance. 

• Although there are coverage issues 
surrounding the use of UI wage- 
record data, they are unlikely to pose 
insurmountable barriers to the 
confident use of this data for 
perfomr»ance assessment. 

• The cost of using UI wage-record 
data can be expected to be much 
lower than current survey costs and 
may actually decline. 

SUBGROUP AND ACTIVITY 
ANALYSES USING UI 
WAGE-RECORD DATA 

The project performed several analyses to 
demonstrate the different ways UI 
wage-record data can be used to examine the 
employment and earnings experien::es of 
JTPA participants. The first set of analyses 
explored the pre-and post-program 
employment and earnings trends of various 
demographic subgroups in the JTPA 
population. The second set of analyses 



considered the experiences of participants 
grouped on the basis of the type of 
interventions they received during their 
tenure in the JTPA program. We examined 
the distribution of participants across 
various sequences of program activities arid 
then looked at the termination outcomes for 
those sequences. Finally, we examined the 
pre-and post-program employment and 
earnings of participants placed into the most 
common JTPA interventions. 

Although our analyses should be viewed 
as exploratory, the findings provide some 
insights into the JTPA program. The major 
findings in these analyses include the 
following: 

• Although the earnings of JTPA 
participants show a sharp decline 
prior to enrollment, employment 
levels remain fairly constant. 

• The average post-program v^amings 
of terminees increase over the 
course of the post-program period, 
bu^ employment levels decline. 

• Terminees with lower pre-program 
earnings and employment 
experiences tend to receive more 
intensive JTPA services. 

• The differentials found in the 
post-program earnings of 
participants receiving different 
types of intervention tend to 
correspond to the differences found 
in their pre-program earnings. 

• There is some evidence that remedial 
and basic education lead to 
increases in employment for 
participants which show up after a 
delay of a few quarters upon 
termination from this activity. 
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The Role of UI 
Wage-Record Data in the 
Context of the 
Performance Standards 
System 

UI wage-record data can play two 
important roles in the context of the JTPA 
performance standards system. The first 
role is to define groups of participants with 
limited employment experience. Once these 
groups are identified, efforts can be made to 
provide performance standards 
adjustments to SDAs based on their levels of 
service to these groups. The evidence 
indicates that such groups can be defined 
and that performance standards 
adjustments are warranted. 

UI wage-record data can also be used to 
develop a variety of post-program 
performance measures. We examined a 
number of candidate measures in this study. 
Of special note is that UI wage-record data 
can be used to capture the level of 
post-program employment retention 
among JTPA participants. 

The major findings to date concerning the 
possible role of UI wage-record data in the 
performance standards system can be 
summarized as follows: 

• The preliminary evidence indicates 
that pre-program UI information can 
effectively be used to isolate 
hard-to-serve client groups and 
provide performance standards 
adjustments to SDAs for serving 
these populations. 

• The time lag involved in gaining 
access to UI wage-record data is not 
viewed as a major barrier to the use 



of this information in the 
perfo Tnance standards system. 

• UI information can be used to design 
a variety of post- program 
performance measures which more 
adequately reflect the goals of the 
Job Training Partnership Act than 
the current survey-based measures, 
e.g., long-term earnings and 
employment retention. 

Implications for the JTPA 
System 

The findings of the first phase of this project 
indicate that JTPA and any other program 
whose goal is to increase the employment 
and earnings of participants can use UI wage 
record data with confidence. Obtaining 
post-program information from state UI 
systems is not only a viable option but far 
more cost-effective than the current practice 
of gathering this information through 
contact with participants. Furthermore, UI 
data is of higher quality than corresponding 
survey-based information on all of the 
criteria we considered. In short, UI wage 
record data can be used to develop an 
extremely flexible, longitudinal database 
representing virtually all JTPA participants, 
which will greatly enhance the atility of the 
Congress to evaluate the performance of the 
program. 

These findings lead to the conclusion that 
further steps should be taken to facilitate the 
movement of JTPA towards a post-program 
assessment system based on UI data. The 
rest of this chapter is devoted to exploring 
the major implications that such a change 
would impose on the JTPA system. This 
change would affect all levels of the 
program, i.e., federal, state, and local. As a 
result, it is necessary to examine the 
implications at each of these levels. 
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FEDERAL LEVEL 

One of the major in\plications of moving tc 
a p>ost-program eissessment « vstem based on 
UI data is the need vo create new 
performance measures. The fact that UI 
information can be used to track indiviciuals 
for multiple quarters after program 
participation offers a great deal of flexibility 
in designing these measures. This was 
illustrated in Chapter 7 where we proposed 
a number of potential Ul-based 
post-program employment and earnings 
measures. 

Of particular importance is the potential 
for developing more effective measures to 
capture the long-term employment and 
earnings experiences of JTPA participants. 
Employment retention among JTPA 
participants is a growing concern of 
Congress as evidenced by a recently 
proposed amendment which calls for 
standards based on "retention for more than 
6 months in unsubsidized employment." 
Although it would be difficult to collect 
accurate and reliable data for an 
employment retention standard in a 
six-month survey of participants, UI wage 
records data can easily and confidently be 
used to develop and implement such a 
standard. 

The introduction of new performance 
measures is not unprecedented in JTPA. In 
fact, only one of the sbc core performance 
measures proposed by DOh for use in PY90 
is a holdover from the original seven 
performance measures. These changes 
reflect DOL's ongoing efforts to improve the 
JTPA performance standards system. Each 
year, these efforts are aided by the advice 
and comments DOL receives from its 
various technical workgroups. These 
workgroups, composed of employment ^i\d 
training professionals from all levels in the 
JTPA system, can be instrumental in the 



development and selection of post-proj^ram 
performance measures based on UI data. 

A second implication of moving to an 
assessment system based on UI data is the 
need to develop a transitional strategy to 
allow states enough time to adjust to the new 
measures. One option is to adopt the 
strategy used to introduce the post-program 
performance me ures based on survey 
data. This strategy allowed each state to 
select from a menu of alternative 
performance standards. This menu 
approach proved to be very effective then, 
and there is no reason to believe that it 
would be less effective now. 

Regardless of the particular strategy 
employed, there should be some flexibility 
to allow states that are currently in a position 
to adopt performance measures based on UI 
data to exercise this option. Assuming the 
eve itual use of such measures throughout 
the system, it seen\s incongruous to prevent 
a state from immediately adopting 
performance measures based on UI data if 
they have the current capacity to do so. 

If a state decides to adopt performance 
measures based on UI data, the state should 
not be required to collect survey data. The 
sole purpose of the surveys is to obtain data 
needed to assess the post-program 
performance of SDAs. States electing to use 
Ul-based measures should not be compelled 
to incur the cost of conducting the survey. 
Depending on how the post-program 
survey is being funded, this cost saving can 
accrue to either the state or the SDAs. 

A final implication concerns the problem 
associated with the time lag with gaining 
access to UI information. DOL confronted 
this issue before in the movement from 
tennination-based performance measures 
to the current post-program measures. The 
solution was to begin the performance 
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measurement period in the last quarter of 
the program year under evaluation. The 
same approach can be used to accommodate 
the greater time lag imposed by using Ul 
wage record information. 

STATE LEVEL 

The first, and most obvious, implication of 
moving to a Ul-based assessment system for 
a state is that the agency administering JTPA 
will need to obtain UI wage record data. 
Many state JTPA agencies, including those 
participating in this project, have already 
developed procedures to gain access to this 
information. However, a number of other 
states will need to forge data exchange 
agreements to obtain the UI data. A state's 
ability to accomplish this task is likely to 
depend on the current level of integration 
and coordination between their JTPA and UI 
systems. In those states where the level of 
contact has been low, the process of 
developing these agreements may foster 
closer working ties between the two 
systems. 

A state v^l also need to decide how to 
address the issue of out-of-state 
employment. There are a number of 
solutions. A state may decide to rely on a 
statistical adjustment procedure. 
Alternatively, it may conduct a 
supplemental post-program survey for 
those terminees who were placed in 
ouNof-state employment. A state may also 
decide to develop cross-state data sharing 
agreements with neighboring states. It is 
worthy to note that there is an increasing 
interest in developing a national archive for 
UI data. If such an archive is developed, the 
coverage issue surrounding out-of-state 
employment will become moot. 

Once access to UI data is obtained, states 
will confront the technical problem of 
incorporating UI information into their 



current JTPA management information 
systems (N4IS). The level of difficulty a state 
experiences will largely depend on the 
current structu^ of its MIS, especially the 
way data are received and stored. 
Furthermore, given the sheer quantity of 
information that can be obtained from state 
UI data systems, decisions must be made 
with respect to the capacity of the state JTPA 
MIS. For example, states will need to decide 
how long they wish to track the progress of 
JTPA terminees given the limits of their MIS 
to store information. 

The movement to a post program 
assessment system based on UI data will 
also induce states to re-examine their 
incentive policies. As in the past, incentive 
policies will have to change to account for 
new measures. Given the possibility of a 
longer time lag between the period of 
performance and the measurement of 
performance, states may also wish to modify 
the procedures they use to distribute 
incentive funds and technical assistance 
money. 

SDA LEVEL 

For many SDAs, DOL's change in PY90 to 
a system that bases adult and welfare 
performance standards solely on 
post-program survey data has created a 
difficult management enviromnent. In the 
past, SDAs have used the performance 
measures to monitor the performance of 
their service providers. However, this 
management tool loses effectiveness when 
performance measures are limited to 
post-program survey data. In the majority 
of states, post-program data is collected 
from only a sample of trie SDA's former 
participants. Consequently, only a very 
small number of terminees served by a given 
subcontractor are likely to be selected for 
post-program follow-up. The number of 
clients sampled from each subcontractor is 
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unlikely to be sufficient to permit confident 
evaluations of subcontractor performance. 
This problem is especially acute in large 
urban SDAs that have an extensive network 
of providers to monitor. 

The movement to an assessment system 
based on UI data would return this 
nianagement tool to SDAs. Through the Ul 
system, post-program infomution can be 
obtained for virtually all of an SDA's 
terminees. An SDA would be able to use this 
information to assess the performance of 
subcontractors on the same measures the 
state uses to assess the SDA's performance. 
The results of these assessments could then 
be used as a basis for future contract 
negotiations. 

Final Implications 



The use of the UI wage record data as an 
evaluation tool has implications beyond the 
JTPA program. Increasingly, Congress has 
turned to the use of performance measures 
to insure greater accountability in federally 
funded programs and to monitor their 
performance. Recent national welfare and 
vocational education legislation are just two 
examples of this trend. In addition, the 
effort to improve the current levels of 
coordination and cooperation among these 
programs is likely to produce higher levels 
of co-enrollment in the fuhire. These two 
factors, performance measures and 
co-enrollment, argue for a common 
framework to measure labor force outcomes 
of employment and training programs. 

Methods of collecting data based on 
surveys are ill-suited to establishing such a 
framework. They require extraordinary 
efforts to impose some consistency among 
the various survey efforts and to avoid 
duplication of effort. Furthermore, each 
program may wish to examine the progress 



of former clients using different 
post-program time frames. This would 
diminish the effectiveness of any 
coordination effort, and many clients who 
are co-enrolled will likely face the prospect 
of being the subject of multiple surveys. 

On the other hand, a coordinated UI data 
collection effort, providing consistent and 
reliable data for all programs, is a distinct 
possibility. Such an effort will allow each 
program to choose its own post-program 
measurement period and still benefit from a 
cost-sharing approach. As federal funds 
become increasingly scarce, greater efforts 
should be made to promote the use of the 
inexpensive and high quality information 
waiting to be accessed on state UI data 
systems. 

Concluding Comments 

Whether to utilize the UI wage record 
system as a basis for monitoring and 
evaluating publicly financed employment 
and training programs is a strategic 
decision. In this decision, there are four 
major issues of concern. The first and most 
basic issue is that of technical feasibility. 
This project has demonstrated that there are 
no insurmountable technical barriers to the 
use of UI wage record data as an evaluation 
tool. Furthermore, the evidence indicates 
that the UI system can provide extremely 
accurate and reliable information on the 
long-term labor force experiences of 
program participants. It can also be the 
source of much needed pre-program 
information. 

The second major issue concerns the costs 
of obtaining the information. An analysis of 
this issue revealed that the UI approach to 
post-program data collection is far more 
cost-effective than the survey based 
methods. This permits more information to 
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be gathered at a much lower cost and 
substantially increases the range of 
evaluative issues that can be addressed. 

The third major concern focuses on 
organizational and procedural issues. This 
study has shown that the extent of the 
problems posed by these issues will vary 
from state to state. However, they do not 
pose insurmountable barriers to the 
adoption of the proposed data collection 
approach. Clear signals from the Congress 
regarding the use of UI wage records for the 
purpose of program evaluation would 



significantly ameliorate remaining concerns 
at the state level and facilitate the necessary 
data exchange agreements. 

Finally, there is the issue of the political will 
required to initialize and implement the 
proposed change. Often there is a resistance 
to change, and nothing short of a 
congressional directive will permit these 
changes to occur. We hope that our 
arguments for the use of UI wage record 
data as an evaluation tool will lead to greater 
support for this approach. 
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Source: United States Department of Labor, Employme . and Training Administration./ 
Unemployment Insurance Service. Comparision of State Unemployment Insurance Laws 
(1989). 1990:725-457/20645. Washington, D.C.: GPO. 
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Table 100. — Definition of Employer 









Nonpr of i t 


Ml othet Employer s-- 




Agr icul t'jra 1 


Domes t ic 




one employee 




10 employees in 


£ I |UUw m a 










20 weeks or 


CQ unless 


one or 


Minimum period 




State 


$20,000 in a CO 


otherwise 


mor ei./ 


of time or 


Alternative 




unless otherwise 


speci f ied 


(21 States) 


payr ol 1 


conditions!/ 




specified 


(6 States) 










(8 States) 












(2) 


(3) 


(4 ) 


(S) 


(6) 


Ala • 








20 weeks 




A laska 








Any time 




Ariz. 5/ 








20 weeks 




Arfc • 




1/ 


X 


10 days 


r^one 


Cal if . 


1 at anytime 




X 


Over $100 in 






and wages in 






qtr . 






excess of SlOO 












in a CqV 










Co Jo. S/ 








Any time 




Conn . 






X 


20 weeks 




Del . 








20 weeks 




DC. 


1 at anytime 


i;500 in CO 


X 


Any time 




Fla. 5/ 


5 in 20 wks or 






20 weeks 






$10,000 in a CQ 


















20 weeks 




Ha wa i 1 




$225 in CQ 


X 


Any cjime 




















employee* 






$300 in qtr . 


Idaho 5/ 






X 


20 weeks 


IIK 








20 weeks 




Ind. 








20 weeks 




Iowa 






X 


20 weeks 




Kans . 








20 weeks 












20 weeks 




Ky. 








20 w<?*»ks 




La . 










Ha ine 




1/ 




20 weeks 





(T.ibl'? ccntinuod on n^^xt p^'^ge 



Table 100, — oof Inltlon of Employer (Continued) 



State 



(1) 



Md. 
Mass . 
Mich. 
Minn. 4/ 



Miss . 
Mo. 
Mont . 
Nebr , 
Nev . 
N.H. 
N.J. 
N. Mex, 
N.y , 
N.C. 1/ 



N. Dak 

Ohio 



Okla . 
Oreg . 
Pa . 
P.R. 

R.I. 



5/ 



Agr icultural 



10 employees in 

20 weeks or 
$20,000 in a CQ 
unless otherwise* 
specified 
(8 States) 



(2) 



4 in 20 wks . or 
$20,000 in a col/ 



6/ 



1 or more at. any 
t ime 

1 or mo r e at .my 
t ime 



Dome St i c 



$1,000 in a 

CQ unless 
otherwise 
spec i C led 
(6 States) 



(3) 



1/ 



6/ 



-1^500 in CO 



$1,000 per 
ind 1 vidual 
or $1,500 
for 2 or 
rf-ot e 



Nonprofit 
Orqani za t ion 



One or 
moreZ^ 
(71 States) 



(4) 



X 
X 
X 



All Other Employers- 
one employee 



M i n imum pe r lod 
of time or 
payrol 1 



(5) 



Any time 
13 weeks 
20 weeks 
20 weeks 



20 weeks 
20 weeks 
Over $1,000 
20 weeks 
$225 in qtr. 
20 weeks 
$1,000 in yr 
20 weeks 
$300 in qtr. 
20 weeks 
?0 weeks 
20 weeks 



in yr 



20 weeks 
18 weeks 
Any time 
Any time 

Any time 



Al te r na 1 1 ve 
conO itions 3/ 



(S) 



$1,000 in CY 



$4S0 in qtr. 



$225 m qt r 



O 



o 



s 

I 

E 

ST. 

§ 



li9 



{TntAr rontinuod on nc>rt. p'lqr ) 
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Tabic 100. — Definition of Employer (Continued) 



State 



(1) 



Agricultural 



10 employees in 

20 veeks or 
20,000 in a CQ 
unless otherwise 
specified 
(8 States) 

(2) 



Domcst ic 



$1,000 in a 
CQ unless 
otherwise 
spec i C led 

(6 State-i) 



(3) 



Nonprof It 
Organ itat ion 



One or 

{21 States) 



(4) 



Ml oth*?r Eiv^ployers- 
one employee 



Hinimum per iod 
of time or 
payed 1 



(S) 



Alter nat ive 
condi tions^/ 



(6) 



S.C. 
S- Dak* 
Tenn. 5/ 

Tex, 

Utah 

vt.V 
va.5./ 
V.I . 

Wash, 
w. Va . 
Wis. 
Wyo, 5/ 



3 in 20 wks. or 
$6^250 in a CQ 



1/ 

1 or more at any 
t ime 
1/ 



$S0O in CO 
1/ 



20 weeks 
20 weeks 
20 weeks 
20 weeks 

$140 in qtr. 
20 weeks 
20 weeks 
Any time 

Any t ime 
20 weeks 
20 weeks 

Over fSOO in yr. 



o 
< 

m 

O 



1/lrclodes other than cash remuneration. 

h^Ml other States cover nonprofit orqan i lat ion>: that employ 4 or more in 20 weeks as required by Federal 
law. 

VOr a quarterly payroll of $1,500, unless otherwi-ie specified. 

VAgr icultural labor performed by an individual 16 yrs. of aqe or younger is excluded froM agricultural 
coverage unless the employer is covered vjnder the F-»deral law, Minn. ; agricultural labor performed by an alien 
in the harvesting of apples is excluded from aqricultural coveraq?, Ma i ne . 

5/states noted exclude alien aqricullural woikprs until J^^nuary 1, 199 3. 

^/coverage on elective basis only, N ■ H. . 
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COVERAGE 



TABLE 101.-*St4tc Coverage Fe«ultlng from changes In Pederal Laws 





Employer 


includes 


any employing 


uni t 


Enpl oymen t 


includes any service 




Liable for any 


Required to 


be 


Liable for 


any 


Required tc tc 


State 


Federal 


tax 


covered under 


Federal tax 


covered unde: 








any 


r eaer a i 


i a w 






FoHpral la^ 


(1) 












( 4 ) 




(S) 


Ala • 


X 




Y 






X 




X 


Alaska 


1/ 




X 






X 




X 


Ariz. 


X 




Y 

A 






X 




X 


ArK • 


X 




Y 






Y 




X 


cai 1 t , 


















Colo ♦ 


Y 




X 






X 




X 


Conn . 


X 




X 






X 




X 


Del • 


X 




X 






X 




X 


D.C* 


1/ 










X 




Y 


Fla • 


X 




X 






X 




X 


Ga, 
















• • • - , . 


Hawa i i 


1 / 










X 




>; 


I daho 












X 




X 


1 11 * 


X 




X 






X 




X 


I nd # 


X 




X 






>: 






I owa 


X 




X 






X 




X 


Kans • 


X 




X 






X 




X 


Ky. 


X 




X 






X 




X 


La • 


X 




X 






X 




X 


nd 1 ne 


X 




X 






X 




X 


KQ • 


X 




X 










* 


Mass • 


















Mich. 


X 














xl/ 


Minn , 


X 




X 






X 




X 


Miss* 












X 




X 


no . 


X 




X 






X 




X 




X 




X 












Neor * 


X 




X 






X 




X 


Nev , 


X 










X 




X 


U IJ 


X y 










X y 






N, J. 


X 




X 










X 


N . Hex , 


X 




X 






X 




X 




















N,C. 


X 




X 






X 




X 


N^Dak . 


X 




X 






X 




X 


Ohio 


X 




X 






X 




x 


Okla. 


X 




X 






X 




X 


Or eg. 












X 




X 


Pa. 


y 










X 




X 


P.R. 


X 




X 






X 






R.I. 


X 




X 






X 




X 





















(Table continued on next page) 
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TABLE 101*--State Coverage Resulting froir Changes in Federal Laws (Cont,) 



State 



(1) 



Employer induces any employing unit 



Liable for any 
Federal tax 



(2) 



Required to be 
cover ed under 
any Federal law 

(3) 



Employment includes any service 



Liable for any 
Federal tax 



(4) 



Required to be 
covered under 
Federal law 

(5) 



5. Dak 
Tenn . 
Tex • 
Utah 
Vt. 
va. 
V.I. 
wash • 
W . Va . 
Wis . 
Wyo. 



X 
X 
X 

X 
X 

X 
X 
X 
X 



1/ 



X 
X 
X 
X 



X 
X 



i/^No such provision; none needed since State law covers employers of one or more 
workers at any tin^e. 

l^Law states that nothing shall be construed tc require identical coverage to 
the FUTA, 

A^Remuner at ion for services performed in the State and subject to the FUTA 
defined as wages for employment. 

i./Not applicar]e to classes of employers whose inclusion would adversely affect 
efficient administration or impair fund Ha ss . ; to service performed by a student in 
a work-study program, or part-time service by a yninor student, or by a memuer of a 
band or orchestra Ki ch , ; or to agricultural labor and domestic service W,Va . , 

-^Elective coverage ujider Stat_^' law for agricultural anc domestic services and 
employers , N^. H . . 
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COVERAGE 

Tab?c 102 * --coverage as Determined by Employer-Employee Relationship 





Ser\'ices 


considered employment unl ess- 






Wor k er s are 


Service is out- 


Wor Ker i s cus- 






free f r om 


s loe r eg u i a r 


tomarily m an 




State 


control over 


\. V \J I S»V \J I L^AO^V? 


1 no epenoe n t 


Other provisions 




P"i it?trnance 


oi empioyer s 


bus 1 ness 








Du $ 1 n e s s 






i 1 ) 




( 3 ) 




(5) 


Ala. 


j 


* • 




Master-ser V Ant . 


A i a s K a 


X 


and X 


and X 




A r ) 2 . 








Ser vice of employee . 


Jk r k 


V 
A 


and X 


and X 


w o ^ A i . 








contract of hire.*./ 




Y 
A 




and X 




Con n • 




and X 


and X 




no 1 


V 

A 


and X 


and X 




U t \- . 








Contract of hire and 


r 1 c . 








master-ser van t .£/ 








Service of empl oyee . 




X 


A n V 

a M U A 


and X 


j-j 3 ^ J 1 


)( 


and X 


and X 




I G 3 h C 


X 




and X 




in. 


X 


and X 


and X 




I no . 


>: 


and X 


and X 




I owa 


A 






Contract of hire,^/ 


V i r c 


A 


a T-i V 

o n a A 














Master-servant ,zj 


T a 


V 
A 


a d X 


;i n H V 




Ti C 1 lit- 


V 
A 


and X 


and X 






X 


and X 


and X 




Kass . 


X 


and X 


and X 




K 1 c h . 


X 






vwiiwiot-w wi nire. - 


K ; n n . 






Y 
A 




Kiss. 


X 






Master^servant - 


Mc . 


X 


and X 


and X 




Kont . 


X 




and X 




Nebr . 


X 


and X 


and X 




Hev. 


X 


and X 


and X 




N .H , 


X 


and X 


and X 




N.J. 


X 


and X 


and X 




N,Mex , 


X 


and X 


and X 




N.Y. 








contract of hire. 2/ 


N.C . 








Contract of hire 










creat ing employee 


N .Dak . 








relationship. 


X 


and X 


and X 


contract of hire. 


Ohio 


X 


and X 


and X 





(Tablt continued on next page) 
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Table 102. — coverage as Deterinined by Ernployer-Employee Relationship (Continued) 



State 

i 1 \ 




services 


cons I dec ed employment un less- 


1 

1 






Wor^;ecs ate 

free frorr. 
contr ol over 
per for inance 

(2) 


Service is out- 
side regular 
course or place 
of employer * s 
business 

(3) 


Worker is cus- 
tomar ily vn an 
independent 
business 

(4) 


nthpr nrovisions 
(5 ) 


Okla. 




















Master-ser vant . 




Or eg . 


X 












and 


X 








ra . 


X 












an'^ 


X 










X 








and 


X 


and 


X 








R.I, 


X 








and 


X 


and 


X 






2/ 














m * m m t 








Contract of hir e 


S.Dak. 


X 












and 


X 








Tenn * 


X 








and 


X 


and 


X 








Tex . 


X 


















CO'^t.rart hire 


Utah 


X 












and 


X 








vt . 


X 








and 


X 


and 


X 








Va . 


X 








and 


X 


and 


X 








V.I. 


X 








and 


X 


and 


X 








Wash . 


X 








and 


X 


and 


X 








w.va . 


X 








\ and 


X 


and 


X 








Wl£ . 


X 












And 


X 








Wyo. 


X 








and 


X 


and 


X 









I'^Service performed by an employee for the person or employing unit employing hir,. 

^''service under any contract of hire, written or era), express or implied. 

i.' By r eg u 1 a 1 1 on . 

i./By judicial interpretation. 
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Table 103. --Signif leant Miscellaneous Bttployment Exclusionsi/ 



State 



(1^ 



Mass, 
Mich, 
Ml nn . 

Miss . 
Mo. 

Mont. 
Nebr . 
Nev. 
N.H. 
N.J, 



Ala . 


X 




X 


Ar 17 . 


X 


Ark, 


X 


Calif . 




Colo. 


X 


Conn. 


X 


Del. 


X 


D.C. 


X 


Fla- 


X 


Ga. 1 


X 



Ace'-.ts on corr- 
m ss ion 



arce 



(2) 



Hawa: ; 


X 


Idaho 


X 


111. 


X 


Ind. 


X 


Iowa 




Kans . 


X 


Ky. 


X 


La. 


> 


Ma 1 ne 


X 


Md, 


X 



N.Mex. 


X 


N.y . 




N.C- 


X 


N.DaK . 


X 


Ohio 


X 


Okla. 


X 


Oreg. 


X 


Pa. 


X 







Real 
estate 



(3) 



X 
X 
X 
X 
X 
X 
X 
X 

X 
X 

X 
X 
X 

X 
X 

X 
X 

2/ 

X 
X 
X 

X 
X 
X 
X 
X 
X 

X 
X 
X 
X 

X 
X 
X 
X 



Casual 
labor 
not in 
course 
of em- 
ployer *s 
business 

(4) 



X 
X 
X 
X 
X 
X 
X 

X 
X 
X 



X 
X 

X 

X 

X 
X 

X 

X 



X 
X 
X 

X 
X 
X 



Part-time 
service for 
nonprofit 
organiza- 
tions exempt 
from Federal 
income taxi/ 

(5) 



X 

X 
X 
X 
X 
X 



X 
X 
X 

X 

X 
X 

X 
X 



Student 

nurses 
and interns 
in employ of 
a hospital 



(6) 



X 
X 
X 



X 
X 
X 

X 
X 
X 
X 



X 
X 
X 
X 



X 
X 
X 
X 
X 



(Table continued on next pa9e} 



Students 
work ing 
for 
schools^/ 
9/10/ 



(7) 



X 
X 
X 
X 
X 
X 
X 
X 

xi/ 

X 

X 
X 
X 
X 
X 
X 
X 
X 
X 
X 

X 
X 
X 
X 

xi/ 

X 
X 
X 
X 
X 

X 
X 
X 
X 
X 
X 
X 
X 
X 
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Table 103. — Significant Miscellaneous Employment Exclusions!/ (Continued) 









casual 


Part-time 


Student 






Agents 


on corn- 


labor 


service for 


nurses 


students 




mission 


not in 


nonpr of it 


and interns 


wor King 








couc se 


organiza- 


in employ of 


tor 


state 


Insur- 


Real 


of em* 


tions exempt 


a hospital 


schools^,'^ 




ance 


estate 


ployer ' s 


from Federal 




9/10/ 








business 


income taxl'^ 






(1) 


(2) 


(3) 


(4 ) 


(5 ) 


(6 ) 


( / ) 






X 


X 


X 




X 


s.c 


X 


X 


X 


X 


X 


X 


S.DaK. 


X 




• • * * 


X 


X 


xl/ 


Tenn. 


X 


X 


* • ■ • 






X 


Tex . 


X 




• fe * * 




X 


X 


Utah 


X 


X 


X 


X 




X 


Vt, 


X 


X 


X 


X 




X 


va. 


X 


X 


X 


X 


X 


X 


V.I . 


« * • 


* • • 


X 






X 


wash . 


X 


X 


X 






X 


w.va. 


X 


* * * 








X 


Wis . 


X 


X 




X 


X 


X 


Wyo . 


» « * 


X 








X 



1/ FOt the major employment exclusions, see text/ sec* 125, 

i/jf the remuneration does not exceed $4S per calendar quarter (or is less than 
S50, in accordance with 1950 amendnent to FUTA ) : in Alaska , S250; Maine ^ Si50, 

1/service in employ of school, college, or university by a student regularly 
enrolled at such institution. 

i/m States noted, law contains broad exclusion of services performed by 
students in the employ of an organization exempt from Federal income tax. D.C> also 
has a provision excluding services performed by a student in the employ of an 
organization exempt from Federal income tax and the remuneration does not exceed SSO 
in a calendar quarter. All but 2 of the States noted, Md . and T^x . , have a 
provision which provides for the coverage of any excluded services which are subject 
to the FUTA. 

^/if the remuneration (exclusive of room, board, and tuition) does not exceed 
$50 per calendar quarter* 

£/ By court decision or attorney general*s opinion. 
VappI icable only while exempt from FUTA. 

i/ooes not exclude such service if performed for a corporation or by industrial 
and debit insurance agents, R.I . 

VaII States except the following exclude service by the spouse of a student in 
the employ of the school: Alaska , Ar k . , Del, , D^C . , Fla , , Hawaii , Idaho , Kans . , 
La,, Maine, M in n, , N*Hex . , Ohio , P .R . / R.l . , Tex., V*l . , and Va. 
^0/a11 States except the following exclude students in work-study programs: 
T>.C., Hawaii I Maine excludes only elementary or secondary school students. 
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TABLE 104. — Exclusions from Service for State and Local Governments 









n vniu ^4 s or 










Le^^*'^'^^^ 5 


i> va ce 


i empui <i t J 


r 01 1 cyniaK 1 ng 




Elec teo 


» n /I ffk A a r c 


na ^ 1 ona i 


AfTtA r 0 A r\ \j 


An/4 &r4\JiCOr\j 


Sta te 


of f ici& 1 s 


01 J uQ I c 1 0 1 y 


vua r 0 ana 


AfftnY ^ vi A 0 £ 

cnip J. Oj f; c 9 










All national 












Guard 






/ 1 \ 




\ ^ 1 






\ 0 ) 


A 1 a • 


V 
A 


V 

A 


y 

A 


y 
A 


V 

A 


X 1 A c Ir A 

A 1 fl 9 N a 


V 

A 


Y 
A 


Y 
A 


y 

A 


y 
A 


A I 1 2 * 


A 


Y 
A 


Y 
A 


y 

A 


V 
A 




A 


Y 
A 


Y 
A 


y 

A 


y 
A 


L> a 1 1 £ . 


Y 


y 

A 


y 
A 


y 
A 
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(Table continued on next page) 
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TABLE 104.— Eicluilon* from S«rvic« for Sttt« and Locul Covernwentt (Continued) 









Hembeta of 










L*di ala tor s 


StAte 


Temporary 


Policymaking 




elected 


and Member a 


National 


emergency 


and Advisory 


State 


of f .ciala 


of judiciary 


Guard and 


enployeet 


positions 








Xir National 












Guard 






(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


Tex* 


X 


X 


X 


X 


X 


Utah 


X 


X 


X 


X 


X 


vt. 


X 


X 


X 


X 


X 


va. 


X 


X 


X 


X 


X 


V,I* 


X 


X 


X 


X 


X 


Wash. 


X 




X 


• ft * » 


X 


w.va. 


X 


X 


X 


X 


X 


Wise*!/ 


X 


X 


X 


X 


X 


Wyo* 


X 


X 


X 


X 


X 



JL/state law does not exclude any of these services. 

l/jn addition to the exclusions listed, excludes official appointed to fill 
unexpired term of elected official, Wis . 
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Appendix B.l 

JTPA Data RIa Elamanta 

(One Record Per Termination) 



Data Item 

1. Qient Identifier 

2. SDA 

3. Title 

4. County of Residence 

5. Date Enrolled in Program 

6. Age 

7. Race/Ethnic Group 

8. Sex 

9. Educational Sutus 

10. Single Head of Household with Dependent under Age 1 8 

1 1 . Presence of Children under 6 

12. Umited English Speaking 

13. Offender 

14. UI Claimant Status 

15. Veteran Status 

1 6. Unemployed 1 5 or More Weeks of Last 26 

17. Employment Status at Enrollment 

18. AFDC Welfare Status 

19. General Assistance Status 

20. Refugee Assistance Status 

21. Tennmation Date 

22. TernUnation Status 

23. Hourly Wage at Termination 

24. Scheduled Weekly Hours at Termination 

25. Post Program Sample Status 

26. Post Program Response Status 

27. Employment Status at 1 3th Week 

28. Weekly Earnings During 13th Week 

29. Weeks Worked in Post Program Period 

Suggested supplemental data itenr»s: 

a) Length of time on welfare 

b> Any additional barriers to employment collected by the State, 

c) Any additional post-program data collected by the State. 
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Appendix B.2 

Ul Data File Elements 

(Multiple Records Per Termination) 



Data Item 

1. Client Identifier 

2. Start Date of Quarter 

3. Employer Identifier 

4. Industry Code 

5. County Code 

6. Weeks Worked During Quarter 

7. Quarterly Earnings 

8. Quarter Index 
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Appendix B.3 

Activity Data File Elennents 

(Multiple Records Per Termination) 



Data Item 

1. Client Identifier 

2. SDA 

3. Title 

4. Date Enrolled in Program 

5. Start Date for Activity 

6. End Date for Activity 

7. Activity Code 

Suggested supplemental data items: 

a) Contact hours for classroom training activities 

b) Occupational Code 

c) Completion Code for Activity 
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